Tag: learning

Two loops, one inside the other

I am developing my skills in programming by attacking the general construct of Markov chains and state space. My theory on the bridging between collective intelligence in human societies and artificial neural networks as simulators thereof is that both are intelligent structures. I assume that they learn by producing many alternative versions of themselves whilst staying structurally coherent, and they pitch each such version against a desired output, just to see how fit that particular take on existence is, regarding the requirements in place.  

Mathematically, that learning-by-doing is a Markov chain of states, i.e. a sequence of complex states, described by a handful of variables, such that each consecutive state in the sequence is a modification of the preceding state, through a logically coherent σ-algebra. My so-far findings suggest that orienting the intelligent structure on specific outcomes, out of all those available, is crucial for the path of learning that structure takes. In other words, the general hypothesis I am sniffing around and digging into is that the way an intelligent structure learns is principally determined by the desired outcomes which the structure is after, more than by the exact basket of inputs it uses. Stands to reason, for a neural network: the thing optimises inputs so as to make it fit to the outcome it seeks to get as close to as possible.  

As I am taking real taste in stepping out of my cavern, I have installed Anaconda on my computer, from https://www.anaconda.com/products/individual/download-success . When I use Anaconda, I use the same JupyterLab online functionality which I have been using so far, with one difference. Anaconda allows me to create a user account with JupyterLab, and to have all my work stored on that account. Probably, there are some storage limits, yet the thing is practical. 

Anyway, I want to program in Python, just as I do it in Excel, intelligent structures able to emulate the collective intelligence of human societies. A basic finding of mine, in the so-far research, is that intelligent structures alter their behaviour significantly depending on the outcome they pursue. The initial landscape I start operating in is akin a junkyard of information. I go to the website of World Bank, for example, I mean the one with freely available data, AKA https://data.worldbank.org , and I start rummaging. Quality of life, size of economies, headcount of populations… What else? Oh, yes, there are things about education, energy consumption and whatnot. All that stuff just piled up nicely, each item easy to retrieve, and yet, how does it all make sense together? My take on the thing is that there is stuff going on, like all the time and everywhere. We are part of that ongoing stuff, actually. Out of that stream of happening, we perceptually single out phenomenological cuts , and we isolate those specific cuts because we are able to measure them with some kind of gauge. Data-driven observation of ourselves in the world is closely connected to our science of measuring and counting stuff. Have you noticed that a basic metric, i.e. how many of us is there around, can take a denominator of one – when we count the population of a city – or a denominator of 10 000, when we are interested in the incidence of criminality. 

Each quantitative variable I can observe and download the dataset of from https://data.worldbank.org  comes out of that complex process of collective cognition, resembling a huge bunch of psychos walking around with rulers and abacuses, trying to measure everything they perceive. I use data as phenomenological description of both the reality those psychos (me included) live in, and the way they measure that reality. I want to check which among those quantitative variables are particularly suitable to represent the things we are really after, our collectively desired outcomes. The method I use to do it consists in producing as many variations of the original dataset as I have variables. Each variation of the original dataset has one variable singled out as output, and the remaining ones are input. I run such variation through a simple neural network – the simpler, the better – where standardised, randomly weighed and neurally activated input gets compared with the pre-set output. I measure the mean expected values of all the variables in such a transformation, i.e. when I run it through 3000 experimental rounds, I measure those means over the same 3000 rounds. I compute the Euclidean distance between each such vector of means and its cousin computed for the original dataset. I assume that, with rigorously the same logical structure of the neural network, those variations differ from each other just by the output variable they are pegged on. When I say ‘pegged’, by the way, I mean that the output variable is not subject to random weighing, i.e. it is not being experimented with. It comes exogenously, and is taken as it is. 

I noticed that each time I do that procedure, with whatever set of variables I take, one or two among them, when taken as output ones, produce variations much closer to the original dataset that other, in terms of Euclidean distance. It looks as if the neural network, when pegged on those particular variables, emulated a process of adaptation particularly similar to what is represented by the original empirical data. 

Now, I want to learn how to program, in Python, the production of alternative ‘input <> output’  couplings out of a source dataset. I already know the general drill for producing just one such coupling. Once I have my dataset read out of a CSV file into a Data Frame in Python Pandas, I start with creating a dictionary of all the numerical columns:

>> dict_numerical = [‘numerical_column1’, ‘numerical_column2’, …, ‘numerical column_n’]

A simple way of doing that, with large data frames, is to type in Python:

>> df.columns

… and it yields a string of labels in quotation marks ‘’, separated with commas. I just copy that lot , without the non-numerical columns, into the square brackets of dict_numerical = […], and Bob’s my uncle. 

Then I make a strictly numerical version of my database, by:

>> df_numerical = pd.DataFrame(df[dict_numerical])

By the way, each time I produce a new data frame, I check its structure with commands ‘df.info()‘ and ‘df.describe()’. At my neophytic level of programming, I want to make sure that what I have in a strictly numerical database is strictly numerical data, i.e. the ‘float64’ type. Here, one hint: when you convert your data from an original Excel file, pay attention to having your decimal point as a point, i.e. as ‘0.0’, not as a comma. With a comma, the Pandas reader tends to interpret such data by default as ‘object’. Annoying. 

Once I have that numerical data frame in place, I make another dictionary of the type:

>> dict_for_Input_pegged_on_X_as_output = [‘numerical_input_column1’, ‘numerical_input_column2’, …, ‘numerical_input_column_k’]

… where k = n -1, of course, and the 1 corresponds to the variable X, supposed to be the output one. 

I use that dictionary to split df_numerical:

>> df_output_X = df_numerical[‘numerical_column_X’]

>> df_input_for_X = df_numerical[dict_for_Input_pegged_on_X_as_output]     

I would like to automatise the process. It means I need a loop. I am looping over a range of numerical columns df_numerical. Let’s dance. I start routinely, in my Anaconda-Jupyter Lab-powered notebook. By the way, I noticed an interesting practical feature of Jupyter Lab. When you start it directly from its website https://jupyter.org , the notebook you can use has somehow limited functionality as compared to the notebook you can create when accessing Jupyter Lab from the Anaconda app on your computer. In the latter case you can create an account with Jupyter Lab, with a very useful functionality of mirroring the content of your cloud account on your hard drive. I know, I know: we use the cloud so as not to collect rubbish on our own disk. Still, Python files are small, they take little space, and I discovered that this mirroring stuff is really useful. 

I open up with importing the libraries I think I will need:

>> import numpy as np

>> import pandas as pd

>> import math

>> import os

As I am learning new stuff, I prefer taking known stuff as my data. Once again, I use a dataset which I made out of Penn Tables 9.1., by kicking out all the rows with empty cells [see: Feenstra, Robert C., Robert Inklaar and Marcel P. Timmer (2015), “The Next Generation of the Penn World Table” American Economic Review, 105(10), 3150-3182, www.ggdc.net/pwt ].

I already have that dataset in my working directory. By the way, when you install Anaconda on a MacBook, its working directory is by default the root directory of the user’s profile. For the moment, I keep ip that way. Anyway, I have that dataset and I read it into a Pandas dataframe:

>> PWT=pd.DataFrame(pd.read_csv(‘PWT 9_1 no empty cells.csv’,header=0))

I create my first dictionaries. I type:

>> PWT.columns

… which yields:

Index([‘country’, ‘year’, ‘rgdpe’, ‘rgdpo’, ‘pop’, ’emp’, ’emp / pop’, ‘avh’,

       ‘hc’, ‘ccon’, ‘cda’, ‘cgdpe’, ‘cgdpo’, ‘cn’, ‘ck’, ‘ctfp’, ‘cwtfp’,

       ‘rgdpna’, ‘rconna’, ‘rdana’, ‘rnna’, ‘rkna’, ‘rtfpna’, ‘rwtfpna’,

       ‘labsh’, ‘irr’, ‘delta’, ‘xr’, ‘pl_con’, ‘pl_da’, ‘pl_gdpo’, ‘csh_c’,

       ‘csh_i’, ‘csh_g’, ‘csh_x’, ‘csh_m’, ‘csh_r’, ‘pl_c’, ‘pl_i’, ‘pl_g’,

       ‘pl_x’, ‘pl_m’, ‘pl_n’, ‘pl_k’],

      dtype=’object’)

…and I create the dictionary of quantitative variables:

>> Variables=[‘rgdpe’, ‘rgdpo’, ‘pop’, ’emp’, ’emp / pop’, ‘avh’,

       ‘hc’, ‘ccon’, ‘cda’, ‘cgdpe’, ‘cgdpo’, ‘cn’, ‘ck’, ‘ctfp’, ‘cwtfp’,

       ‘rgdpna’, ‘rconna’, ‘rdana’, ‘rnna’, ‘rkna’, ‘rtfpna’, ‘rwtfpna’,

       ‘labsh’, ‘irr’, ‘delta’, ‘xr’, ‘pl_con’, ‘pl_da’, ‘pl_gdpo’, ‘csh_c’,

       ‘csh_i’, ‘csh_g’, ‘csh_x’, ‘csh_m’, ‘csh_r’, ‘pl_c’, ‘pl_i’, ‘pl_g’,

       ‘pl_x’, ‘pl_m’, ‘pl_n’, ‘pl_k’]

The ‘Variables’ dictionary serves me to mutate the ‘PWT’ dataframe into its close cousin, obsessed with numbers, namely into ‘PWT_Numerical’:

>> PWT_Numerical = pd.DataFrame(PWT[Variables])

I quickly check the PWT_Numerical’s driving licence, by typing ‘PWT_Numerical.info()’ and  ‘PWT_Numerical.shape’. All is well, data is in the ‘float64’ format, there are 42 columns and 3006 rows, the guy is cleared to go.

Once I have that nailed down, I mess around a bit with creating names for my cloned datasets. I practice with the following loop:

>> for i in range(42):

    print(“Input_for_”+PWT_Numerical.iloc[:,i].name) 

It yields a list of names for input databases in various ‘input <> output’ configurations of my experiment with the PWT 9.1 dataset. The ‘print’ command gives a string of 42 names: Input_for_rgdpe, Input_for_rgdpo, Input_for_pop etc. 

In my next step, I want to make that outcome durable. The ‘print’ command just prints the output of the loop, it does not store it in any logical structure. The output is gone as soon as it is printed. I create a loop that makes a dictionary, this time with names of output data frames:

>> Names_Output_Data=[] # Here, I create an empty dictionary

>> for i in range(42): # I design the loop

    >> Name_Output_Data=PWT_Numerical.iloc[:,i].name # I create a mechanism for generating strings to fill the dictionary up. 

    >> Names_Output_Data.append(Name_Output_Data) # This is the mechanism of appending   the dictionary with names generated in the previous command 

I check the result by typing the name of the dictionary – ‘Names_Output_Data’ – and executing (Shift + Enter in Jupyter Lab). It yields a full dictionary, filled with column names from PWT_Numerical. 

Now,  pass to designing my Markov chain of states, i.e. into making an intelligent structure, which produces many alternative versions of itself and tests them for fitness to meet a pre-defined desired outcome. In my neophyte’s logic, I see it as two loops, one inside the other. 

The big, external loop is the one which clones the initial ‘PWT_Numerical’ into pairs of data frames of the style: ’Input variables’ plus ‘Output variable’. I make as many such cloned pairs as there are numerical variables in PWT_Numerical, i.e. 42. Thus, my loop opens up as ‘for i in range(42):’. Inside each iteration of that loop, there is an internal loop of passing the input variables  through a very simple perceptron, assessing the error in estimating the output variable, and then feeding the error forward. Now, I will present below the entire code for those two loops, and then discuss what works, what doesn’t, and what I have no idea how to check whether it works or not. The code is grammatically correct in Python, i.e. it does not yield any error message when put to execution (Shift + Enter in JupyterLab, by the way).  After I present the entire code, I will discuss, further below, its particular parts. Anyway, here it is:

>> List_of_Output_DB=[]

>>Names_Output_Data=[]

>>MEANS=[]

>> Source_means=np.array(PWT_Numerical.mean())

>> EUC=[]

>>for i in range(42):

    >> Name_Output_Data=PWT_Numerical.iloc[:,i].name

    >> Names_Output_Data.append(Name_Output_Data)

    >> Output=pd.DataFrame(PWT_Numerical.iloc[:,i])    

    >> Mean=Output.mean()

   >> MEANS.append(Mean)

    >> Input=pd.DataFrame(PWT_Numerical.drop(Output,axis=1)) 

   >> Input_STD=pd.DataFrame(Input/Input.max(axis=0))

    >> ER=[]

    >> Transformed=[]

      >> for j in range(30):        

>> Input_STD_randomized=Input.iloc[j]*np.random.rand(41)

        >> Input_STD_summed=Input_STD_randomized.sum(axis=0)

        >> T=math.tanh(Input_STD_summed)

        >> D=1-(T**2)

        >> E=(Output.iloc[j]-T)*D

        >> E_vector=np.array(np.repeat(E,41))        

>> Next_row_with_error=Input_STD.iloc[j+1]+E_vector

>> Next_row_DESTD=Next_row_with_error*Input.max(axis=0)

        >> ER.append(E)

        >> ERROR=pd.DataFrame(ER)

        >> Transformed.append(Next_row_DESTD)

        >> CLONE=pd.DataFrame(Transformed).mean()

>> frames=[CLONE,MEANS[i]]

>> CLONE_Means=np.array(pd.concat(frames))

>> Euclidean=np.linalg.norm(Source_means-CLONE_Means)

>> EUC.append(Euclidean)

>> print(‘Finished’)   

Here is a shareable link to my Python file with that code inside: http://localhost:8880/lab/tree/Practice%20Dec%208%202020.ipynb  . I hope it works. 

I start explaining this code casually, from its end. This is a little trick I discovered as regards looping on datasets. Looping takes time and energy. In my struggles to learn Python, I have already managed to make a loop which kept looping forever. All I did was to call the loop as ‘for i in range PWT.index:’, without putting any ‘break’ command at the end. Yes, the index of a data frame is a finite number, yet it is also a sequence. When you don’t break explicitly the looping over that sequence, it will loop over and over again. 

Anyway, the trick. I put the command ‘print(‘Finished’)’ at the very end of the code, after all the loops. When the thing is done with being an intelligent structure at work, it simply prints ‘Finished’ in the next line. Among other things, it allows me to count the time it needs to deal with a given amount of data. As you might have already noticed, whilst I have a dataset with index = 3005 rows, I made the internal loop of the code to go just over 30 rows: ‘for j in range (30)’. The code took some 4 seconds in total to create 42 big loops (‘for i in range (42)’) , and then to loop over 30 rows of data inside each of them. It gives like 42*30 = 1260 experimental rounds in 10 seconds, thus something like 0,0079 seconds per one round. If I took the full dataset of 3005 rows, it would be like 42*3000*0,0079 = 1000 seconds, i.e. 16,6666 minutes. Satanic. I like it. 

Before opening each level of looping, I create empty lists. You can see:

>> List_of_Output_DB=[]

>>Names_Output_Data=[]

>>MEANS=[]

>> Source_means=np.array(PWT_Numerical.mean())

>> EUC=[]

… before I open the external loop, and…

  >> ER=[]

>> Transformed=[]

… before the internal loop.

I noticed that I create those empty lists in a loop, essentially. This is more than just a play on words. When I code a loop, I have output of the loop. The loop does something, and as it does, I discover I want to store that particular outcome in some kind of repository vessel, and I go back to the lines of code before the loop opens and I add an empty list, just in case. I come up with a smart name for the   list, e.g. MEANS, which stands for the mean values of numerical variables, such as they are after being transformed by the perceptron. Mathematically, it is the most basic representation of expected state in a particular transformation of the source dataset “PWT’. 

I code it as ‘MEANS=[]’, and, once I have done that, I add a mechanism of updating a list, inside the loop. This, in turn, goes in two steps. First, I code the variable which should be stored in that list. In the case of ‘MEANS’, as this list is created before I open the big loop of 42 ‘input <> output’ mutations, I append it in that loop. Logically, is must be appended with the mean expected values of output variables in each instance of the big loop. I code it in the big loop, and before opening the internal loop, as:

>> Output=pd.DataFrame(PWT_Numerical.iloc[:,i])  # Here, I define the data frame for the output variable in this specific instance of the big loop   

>> Mean=Output.mean() # Now, I define the function to generate values, which later append the ‘MEANS’ list

    >> MEANS.append(Mean) # Finally, I append the ‘MEANS’ list with values generated in the previous line of the code. 

It is a good thing for me to write about the things I do. I have just noticed that I use two different methods of storing partial outcomes of my loops. The first one is the one I have just presented. The second one is visible in the part of code presented below, included in the internal loop ‘for j in range(number of rows experimented with)’, range(30) in the occurrence tested. 

In this situation, I need to store in some kind of repository the values of input variables transformed by the neural network, i.e. with local error from each experimental round fed forward to the next experimental round. I need to store the way my data looks under each possible orientation of the intelligent structure I assume it represents. I denote that data under the general name ‘Transformed’, and, before opening the internal loop, just at the end of the big external loop, I define an empty list: ‘Transformed=[]’, which is supposed to contain those values I want.

In other words, when I structure the big external loop, I go like: 

# Step 1: for each variables in the dataset, i.e. ‘for i in range(number of variables)’, split the overall dataset into into this variable as the output, in a separate data frame, and all the other variables grouped separately as input. These are the lines of code:

>> Output=pd.DataFrame(PWT_Numerical.iloc[:,i])  # I define the output variable 

[…]    

>> Input=pd.DataFrame(PWT_Numerical.drop(Output,axis=1)) # I drop the output from the entire dataset and I group the remaining columns as ‘Input’

# Step 2: I standardise the input data by denominating it over the respective maximums for each variable:    

>> Input_STD=pd.DataFrame(Input/Input.max(axis=0))

# Step 3: I define, at the end of the big external loop, containers for data which I want to store from each round of the big loop:

>> ER=[] # This is the list of local errors generated by the perceptron when working with each ‘Input <> Output’ configuration

    >> Transformed=[] # That’s the container for input data transformed by the perceptron 

# Step 4: I open the internal loop, with ‘for j in range(number of rows to experiment with)’, and I start by coding the computational procedure of the perceptron:

>> Input_STD_randomized=Input.iloc[j]*np.random.rand(41) # I weigh each empirical, standardised value in this specific row with a random weight

        >> Input_STD_summed=Input_STD_randomized.sum(axis=0) # I sum the randomised values from that specific row of input. This line of code together with the preceding one are equivalent to the mathematical structure ‘∑x*random’.

        >> T=math.tanh(Input_STD_summed) # I compute the hyperbolic tangent of summed, randomised input data

        >> D=1-(T**2) # I compute the local first derivative of the hyperbolic tangent

        >> E=(Output.iloc[j]-T)*D # I compute the error, as: (Expected Output minus Hyperbolic Tangent of Randomised Input) times local derivative of the Hyperbolic Tangent

        >> E_vector=np.array(np.repeat(E,41)) # I create a NumPy array, with the error repeated as many times as there are input variables.

>> Next_row_with_error=Input_STD.iloc[j+1]+E_vector # I feed the error forward. In the next experimental row ‘j+1’, error from row ‘j’ is added to the value of each standardised input variable. This is probably the most elementary representation of learning: I include into my input for the next go the knowledge about what I f**ked up in the previous go. This line creates the transformed input data I want to store later on. 

# Step 5: I collect and store information about the things my perceptron did to input data in the given j-th round of the internal loop:

>> Next_row_DESTD=Next_row_with_error*Input.max(axis=0) # I destandardise the data transformed by the perceptron. It is like translating the work of the perceptron, which operates on standardised values, back into the measurement scale proper to each variable. In a sense, I deneuralise that data. 

        >> ER.append(E) # I collect and store error in the ER list

        >> ERROR=pd.DataFrame(ER) #I transform the ER list into a data frame, which I name ‘ERROR’. I do it a few times with different data, and, quite honestly, I do it intuitively. I already know that data frames defines in Pandas are somehow handier to do statistics with than lists defined in the basic code of Python. Just as honestly: I know too little yet about programming to know whether this turn of code makes sense at all.   

        >> Transformed.append(Next_row_DESTD) # I collect and store the destandardized, transformed input data in the ‘Transformed’ list.

# Step 6: I step out of both loops, and I start putting some order in the data I generated and collected. Stepping out of both loops means that in my code, the lines presented below have no indent. They all start at the left margin, just as the definition of the big external loop.

       >> CLONE=pd.DataFrame(Transformed).mean() # I transform the ‘Transformed’ list into a data frame. Same procedure as two lines of code earlier, only now, I know why I do it. I intend to put together the mean values of destandardised input with the mean value of output, and I am going to do it by concatenation of data frames. 

    >> frames=[CLONE,MEANS[i]] # I define the data frames for concatenation. I put together mean values in the input variables, generated in this specific, i-th round of the big external loop, with the mean value of the output variable corresponding to the same i-th round. You can notice that in the full code, such as I presented it earlier in this update, at this verse of code I move back by one indent. In other words, this definition is already outside of the internal loop, and still inside the big external loop. 

    >> CLONE_Means=np.array(pd.concat(frames)) # I concatenate the data I defined in the previous line. 

    >> Euclidean=np.linalg.norm(Source_means-CLONE_Means) # Something I need for my science. I estimate the mathematical similarity between the source data set ‘PWT_Numerical’, and the data set created by the perceptron, in the given i-th round of the big external loop. I do it by computing the Euclidean distance between the respective vectors of mean expected values in this specific pair of datasets, i.e. the pair ‘source vs i-th clone’.

    >> EUC.append(Euclidean) # I collect and store information generated in the ‘Euclidean’ line. I store it in the EUC list, which I opened as empty before starting the big external loop. 

One step out of the cavern

I have made one step further in my learning of programming. I finally have learn’t at least one method of standardising numerical values in a dataset. In a moment, I will show what exact method did I nail down. First, I want to share a thought of more general nature. I learn programming in order to enrich my research on the application of artificial intelligence for simulating collective intelligence in human societies. I have already discovered the importance of libraries, i.e. ready-made pieces of code, possible to call with a simple command, and short-cutting across many verses of code which I would have to write laboriously. I mean libraries such as NumPy, Pandas, Math etc. It is very similar to human consciousness. Using pre-constructed cognitive structures, i.e. using language and culture is a turbo boost for whatever we do of things that humans are supposed to do when being a civilisation.  

Anyway, I kept working with the dataset which I had already mentioned in my earlier updates, namely a version of Penn Tables 9.1., cleaned of all the rows with empty cells [see: Feenstra, Robert C., Robert Inklaar and Marcel P. Timmer (2015), “The Next Generation of the Penn World Table” American Economic Review, 105(10), 3150-3182, www.ggdc.net/pwt ]. Thus I started by creating an online notebook at JupyterLab (https://jupyter.org/try), with Python 3 as its kernel. Then I imported what I needed from Python in terms of ready-cooked culture, i.e. I went:

>> import numpy as np

>> import pandas as pd

>> import os

I uploaded the ‘PWT 9_1 no empty cells.csv’ file from my computer, and, just in case, I checked its presence in the working directory, with >> os.listdir(). I read the contents of the file into a Pandas Data Frame, which spells: PWT = pd.DataFrame(pd.read_csv(‘PWT 9_1 no empty cells.csv’)). Worked.  

In my next step, as I planned to mess up a bit with the columns of that dataset, I typed: PWT.columns. The thing nicely gave me back a list of columns, i.e. literally a list of labels in quotation marks [‘’]. I used that list to create a dictionary of columns with numerical values, and therefore the most interesting to me. I went:

>> Variables=[‘rgdpe’, ‘rgdpo’, ‘pop’, ’emp’, ’emp / pop’, ‘avh’,

       ‘hc’, ‘ccon’, ‘cda’, ‘cgdpe’, ‘cgdpo’, ‘cn’, ‘ck’, ‘ctfp’, ‘cwtfp’,

       ‘rgdpna’, ‘rconna’, ‘rdana’, ‘rnna’, ‘rkna’, ‘rtfpna’, ‘rwtfpna’,

       ‘labsh’, ‘irr’, ‘delta’, ‘xr’, ‘pl_con’, ‘pl_da’, ‘pl_gdpo’, ‘csh_c’,

       ‘csh_i’, ‘csh_g’, ‘csh_x’, ‘csh_m’, ‘csh_r’, ‘pl_c’, ‘pl_i’, ‘pl_g’,

       ‘pl_x’, ‘pl_m’, ‘pl_n’, ‘pl_k’]

The ‘Variables’ dictionary served me to make a purely numerical mutation of my dataset, namely: PWTVar=pd.DataFrame(PWT[Variables]).  

I generated the fixed components of standardisation in my data, i.e. maximums, means, and standard deviations across columns in PWTVar. It looked like this: 

>> Maximums=PWTVar.max(axis=0)

>> Means=PWTVar.mean(axis=0)

>> Deviations=PWTVar.std(axis=0)

The ‘axis=0’ part means that I want to generate those values across columns, not rows. Once that done, I made my two standardisations of data from PWTVar, namely: a) standardisation over maximums, like s(x) = x/max(x) and b) standardisation by mean-reversion, where s(x) = [x – avg(x)]/std(x)]. I did it as:

>> Standardized=pd.DataFrame(PWTVar/Maximums)

>> MR=pd.DataFrame((PWTVar-Means)/Deviations)

I used here the in-built function of Python Pandas, i.e. the fact that they automatically operate data frames as matrices. When, for example, I subtract ‘Means’ from ‘PWTVar’, the one-row matrix of ‘Means’ gets subtracted from each among the 3005 rows of ‘PWTVar’ etc. I checked those two data frames with commands such as ‘df.describe()’, ’df.shape’, and df.info(), just to make sure they are what I think they are. They are, indeed. 

Standardisation allowed me to step out of my cavern, in terms of programming artificial neural networks. The next step I took was to split my numerical dataset PWTVar into one output variable, on the one hand, and all the other variables grouped as input. As output, I took a variable which, as I have already found out in my research, is extremely important in social change seen through the lens of Penn Tables 9.1. This is ‘avh’ AKA the average number of hours worked per person per year. I did:  

>> Output_AVH=pd.DataFrame(PWTVar[‘avh’])

>> Input_dict=[‘rgdpe’, ‘rgdpo’, ‘pop’, ’emp’, ’emp / pop’, ‘hc’, ‘ccon’, ‘cda’,

        ‘cgdpe’, ‘cgdpo’, ‘cn’, ‘ck’, ‘ctfp’, ‘cwtfp’, ‘rgdpna’, ‘rconna’,

        ‘rdana’, ‘rnna’, ‘rkna’, ‘rtfpna’, ‘rwtfpna’, ‘labsh’, ‘irr’, ‘delta’,

        ‘xr’, ‘pl_con’, ‘pl_da’, ‘pl_gdpo’, ‘csh_c’, ‘csh_i’, ‘csh_g’, ‘csh_x’,

        ‘csh_m’, ‘csh_r’, ‘pl_c’, ‘pl_i’, ‘pl_g’, ‘pl_x’, ‘pl_m’, ‘pl_n’,

        ‘pl_k’] 

#As you can see, ‘avh’ is absent from the ‘Input-dict’ dictionary 

>> Input = pd.DataFrame(PWT[Input_dict])

The last thing that worked, in this episode of my learning, was to multiply the ‘Input’ dataset by a matrix of random float values generated with NumPy:

>> Randomized_input=pd.DataFrame(Input*np.random.rand(3006,41)) 

## Gives an entire Data Frame of randomized values

It works again

I intend to work on iterations. My general purpose with learning to program in Python is to create my own algorithms of artificial neural networks, in line with what I have already done in that respect using just Excel. Iteration is the essence of artificial intelligence, to the extent that the latter manifests as an intelligent structure producing many alternative versions of itself. Many means one at a time over many repetitions. 

When I run my neural networks in Excel, they do a finite number of iterations. That would be a Definite Iteration in Python, thus the structure based on the ‘for’ expression. I am helping myself  with the tutorial available at https://realpython.com/python-for-loop/ . Still, as programming is supposed to enlarge my Excel-forged intellectual horizons, I want to understand and practice the ‘while’ loop in Python, thus Indefinite Iteration (https://realpython.com/python-while-loop/ ).

Anyway, programming a loop is very different from looping over multiple rows of an Excel sheet. The latter simply makes a formula repeat over many rows, whilst the former requires defining the exact operation to iterate, the input domain which the iteration takes as data, and the output dataset to store the outcome of iteration.

It is time, therefore, to describe exactly the iteration I want to program in Python. As a matter of fact, we are talking about a few different iterations. The first one is the standardisation of my source data. I can use two different ways of standardising it, depending on the neural activation function I use. The baseline method is to standardise each variable over its maximum, and then it fits every activation function I use. It is standardised value of x, AKA s(x), being calculated as s(x) = x/max(x). 

If I focus just on the hyperbolic tangent as activation function, I can use the first method, or I can standardise by mean-reversion, where s(x) = [x – avg(x)]/std(x). In a first step, I subtract from x the average expected value of x – this is the the [x – avg(x)] expression – and then I divide the resulting difference by the standard deviation of x, or std(x). 

The essential difference between those two modes of standardisation is the range of standardised values. When denominated in units of the max(x), standardised values range in 0 ≥ std(x) ≥ 1. When I standardise by mean-reversion, I have -1 ≥ std(x) ≥ 1.

The piece of programming I start that specific learning of mine with consists in transforming my source Data Frame ‘df’ into its standardised version ’s_df’ by dividing values in each column of df by their maximums. As I think of all that, it comes to my mind what I have recently learnt, namely that operations on Numpy arrays, in Python, are much faster than the same operations on data frames built with Python Pandas. I check if I can make a Data Frame out of an imported CSV file, and then turn it into a Numpy array. 

Let’s walse. I start by opening JupyterLab at https://hub.gke2.mybinder.org/user/jupyterlab-jupyterlab-demo-nocqldur/lab and creating a notebook with Python 3 as its kernel. Then, I import the libraries which I expect to use one way or another: NumPy, Pandas, Matplot, OS, and Math. In other words, I do:

>> import numpy as np

>> import pandas as pd

>> import matplotlib.pyplot as plt

>> import math

>> import os

Then, I upload a CSV file and I import it into a Data Frame. It is a database I used in my research on cities and urbanization, its name is ‘DU_DG database.csv’, and, as it is transformed from an Excel file, I take care to specify that separators are semi-columns.

>> DU_DG=pd.DataFrame(pd.read_csv(‘DU_DG database.csv’,sep=’;’)) 

The resulting Data Frame is structured as:

Index([‘Index’, ‘Country’, ‘Year’, ‘DU/DG’, ‘Population’,

       ‘GDP (constant 2010 US$)’, ‘Broad money (% of GDP)’,

       ‘urban population absolute’,

       ‘Energy use (kg of oil equivalent per capita)’, ‘agricultural land km2’,

       ‘Cereal yield (kg per hectare)’],

      dtype=’object’)

Import being successful (I just check with commands ‘DU_DG.shape’ and ‘DU_DG.head()’), I am trying to create a NumPy array. Of course, there is not much sense in translating names of countries and labels of years into a NumPy array. I try to select numerical columns ‘DU/DG’, ‘Population’, ‘GDP (constant 2010 US$)’, ‘Broad money (% of GDP)’, ‘urban population absolute’, ‘Energy use (kg of oil equivalent per capita)’, ‘agricultural land km2’, and ‘Cereal yield (kg per hectare)’, by commanding:

>> DU_DGnumeric=np.array(DU_DG[‘DU/DG’,’Population’,’GDP (constant 2010 US$)’,’Broad money (% of GDP)’,’urban population absolute’,’Energy use (kg of oil equivalent per capita)’,’agricultural land km2′,’Cereal yield (kg per hectare)’]) 

The answer I get from Python 3 is a gentle ‘f**k you!’, i.e. an elaborate error message. 

KeyError                                  Traceback (most recent call last)

/srv/conda/envs/notebook/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)

   2656             try:

-> 2657                 return self._engine.get_loc(key)

   2658             except KeyError:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: (‘DU/DG’, ‘Population’, ‘GDP (constant 2010 US$)’, ‘Broad money (% of GDP)’, ‘urban population absolute’, ‘Energy use (kg of oil equivalent per capita)’, ‘agricultural land km2’, ‘Cereal yield (kg per hectare)’)

During handling of the above exception, another exception occurred:

KeyError                                  Traceback (most recent call last)

<ipython-input-18-e438a5ba1aa2> in <module>

—-> 1 DU_DGnumeric=np.array(DU_DG[‘DU/DG’,’Population’,’GDP (constant 2010 US$)’,’Broad money (% of GDP)’,’urban population absolute’,’Energy use (kg of oil equivalent per capita)’,’agricultural land km2′,’Cereal yield (kg per hectare)’])

/srv/conda/envs/notebook/lib/python3.7/site-packages/pandas/core/frame.py in __getitem__(self, key)

   2925             if self.columns.nlevels > 1:

   2926                 return self._getitem_multilevel(key)

-> 2927             indexer = self.columns.get_loc(key)

   2928             if is_integer(indexer):

   2929                 indexer = [indexer]

/srv/conda/envs/notebook/lib/python3.7/site-packages/pandas/core/indexes/base.py in get_loc(self, key, method, tolerance)

   2657                 return self._engine.get_loc(key)

   2658             except KeyError:

-> 2659                 return self._engine.get_loc(self._maybe_cast_indexer(key))

   2660         indexer = self.get_indexer([key], method=method, tolerance=tolerance)

   2661         if indexer.ndim > 1 or indexer.size > 1:

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/index.pyx in pandas._libs.index.IndexEngine.get_loc()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

pandas/_libs/hashtable_class_helper.pxi in pandas._libs.hashtable.PyObjectHashTable.get_item()

KeyError: (‘DU/DG’, ‘Population’, ‘GDP (constant 2010 US$)’, ‘Broad money (% of GDP)’, ‘urban population absolute’, ‘Energy use (kg of oil equivalent per capita)’, ‘agricultural land km2’, ‘Cereal yield (kg per hectare)’)

Didn’t work, obviously. I try something else. I proceed in two steps. First, I create a second Data Frame out of the numerical columns of DU_DG. I go:

>> DU_DGNumCol=pd.DataFrame(DU_DG.columns[‘DU/DG’, ‘Population’,’GDP (constant 2010 US$)’, ‘Broad money (% of GDP)’,’urban population absolute’,’Energy use (kg of oil equivalent per capita)’, ‘agricultural land km2’,’Cereal yield (kg per hectare)’])

Python seems to have accepted the command without reserves, and yet something strange happens. Informative commands about that second Data Frame, i.e. DU_DGNumCol, such as ‘DU_DGNumCol.head()’, ‘DU_DGNumCol.shape’ or ‘DU_DGNumCol.info‘ don’t work, as if DU_DGNumCol had no structure at all.

Cool. I investigate. I want to check how does Python see data in my DU_DG data frame. I do ‘DU_DG.describe()’ first, and, to my surprise, I can see descriptive statistics just for columns ‘Index’ and ‘Year’. The legitimate WTF? question pushes me to type ‘DU_DG.info()’ and here is what I get:

<class ‘pandas.core.frame.DataFrame’>

RangeIndex: 896 entries, 0 to 895

Data columns (total 11 columns):

Index                                           896 non-null int64

Country                                         896 non-null object

Year                                            896 non-null int64

DU/DG                                           896 non-null object

Population                                      896 non-null object

GDP (constant 2010 US$)                         896 non-null object

Broad money (% of GDP)                          896 non-null object

urban population absolute                       896 non-null object

Energy use (kg of oil equivalent per capita)    896 non-null object

agricultural land km2                           896 non-null object

Cereal yield (kg per hectare)                   896 non-null object

dtypes: int64(2), object(9)

memory usage: 77.1+ KB

I think I understand. My numerical data has been imported as object, and I want it to be float values.  Once again, I have the same valuable lesson: before I do anything with my data, in Python, I need to  check and curate it. It is strangely connected to my theory of collective intelligence. Our human perception accepts empirical experience for further processing, especially for collective processing at the level of culture, only if said experience has the right form. We tend to ignore phenomena, which manifest in a form we are not used to process cognitively.

Just by sheer curiosity, I take another dataset and I repeat the whole sequence of import from CSV, and definition of data type. This time, I take a reduced version of Penn Tables 9.1. The full citation due in this case is: Feenstra, Robert C., Robert Inklaar and Marcel P. Timmer (2015), “The Next Generation of the Penn World Table” American Economic Review, 105(10), 3150-3182, available for download at www.ggdc.net/pwt. The ‘reduced’ part means that I took out of the database all the rows (i.e. country <> year observations) with at least one empty cell. I go:

>> PWT=pd.DataFrame(pd.read_csv(‘PWT 9_1 no empty cells.csv’)) 

…aaaaand it lands. Import successful. I test the properties of PWT data frame:

>> PWT.info()

yields:

<class ‘pandas.core.frame.DataFrame’>

RangeIndex: 3006 entries, 0 to 3005

Data columns (total 44 columns):

country      3006 non-null object

year         3006 non-null int64

rgdpe        3006 non-null float64

rgdpo        3006 non-null float64

pop          3006 non-null float64

emp          3006 non-null float64

emp / pop    3006 non-null float64

avh          3006 non-null float64

hc           3006 non-null float64

ccon         3006 non-null float64

cda          3006 non-null float64

cgdpe        3006 non-null float64

cgdpo        3006 non-null float64

cn           3006 non-null float64

ck           3006 non-null float64

ctfp         3006 non-null float64

cwtfp        3006 non-null float64

rgdpna       3006 non-null float64

rconna       3006 non-null float64

rdana        3006 non-null float64

rnna         3006 non-null float64

rkna         3006 non-null float64

rtfpna       3006 non-null float64

rwtfpna      3006 non-null float64

labsh        3006 non-null float64

irr          3006 non-null float64

delta        3006 non-null float64

xr           3006 non-null float64

pl_con       3006 non-null float64

pl_da        3006 non-null float64

pl_gdpo      3006 non-null float64

csh_c        3006 non-null float64

csh_i        3006 non-null float64

csh_g        3006 non-null float64

csh_x        3006 non-null float64

csh_m        3006 non-null float64

csh_r        3006 non-null float64

pl_c         3006 non-null float64

pl_i         3006 non-null float64

pl_g         3006 non-null float64

pl_x         3006 non-null float64

pl_m         3006 non-null float64

pl_n         3006 non-null float64

pl_k         3006 non-null float64

dtypes: float64(42), int64(1), object(1)

memory usage: 1.0+ MB

>> PWT.describe()

gives nice descriptive statistics. This dataset has been imported in the format I want. I do the same thing I attempted with the DU_DG dataset: I try to convert it into a NumPy array and to check the shape obtained. I do:

>> PWTNumeric=np.array(PWT)

>> PWTNumeric.shape

I get (3006,44), i.e. 3006 rows over 44 columns. 

I try to wrap my mind around standardising values in PWT. I start gently. I slice one column out of PWT, namely the AVH variable, which stands for the average number of hours worked per person per year. I do:

>> AVH=pd.DataFrame(PWT[‘avh’])

>> stdAVH=pd.DataFrame(AVH/AVH.max())

Apparently, it worked. I check with ‘stdAVH.describe()’ and I get a nice distribution of values between 0 and 1. 

I do the same thing with mean-reversion. I create the ‘mrAVH’ data frame according to the s(x) = [x – avg(x)]/std(x) drill. I do:

 >> mrAVH=pd.DataFrame((AVH-AVH.mean())/AVH.std())

…and I get a nice distribution of mean reverted values. 

Cool. Now, it is time to try and iterate the same standardisation over many columns in the same Data Frame. I have already rummaged a bit and apparently it is not going to as simple as in Excel. It usually isn’t.

That would be all in that update. A short summary is due. It works again. I mean, learning something and keeping a journal of how exactly I learn, that thing works. I feel that special vibe, like ‘What the hell, even if it sucks, it is interesting’. Besides the technical details of programming, I have already learnt two big things about data analysis in Python. Firstly, however comfortable it is to use libraries such as NumPy or Pandas, being really efficient requires the understanding of small details at the very basic level, e.g. conversion of data types, and, as a matter of fact, the practical workability of different data types, selection of values in a data set, by row and by column, iteration over rows and columns etc. Secondly, once again, data works well in Python when it has been properly curated prior to analysis. Learning quick algorithmic ways to curate that data, without having to do is manually in Excel, is certainly an asset, which I need to learn.  

Making my brain dance to a slightly different tune

 

I have those many different things running in parallel. I believe it is called ‘living’. I am trying to handle that complexity without inventing special metaphysics for that occasion, like ‘It is pointless to do X, I should entirely focus on Y and Z’. Over that half of a century that I have been in this world, I have come to appreciate the fact that my individual existence is truly a complex singularity. What is pointless is the denial of that fact.

Anyway, I keep nailing down two things: my business plan for the EneFin project, and my teaching programmes. As for the business plan, I made a big step forward in putting all its pieces together in my last update in French, namely « Deux cerveaux, légèrement différents l’un de l’autre ». Now, I am going to do something I like doing, i.e. reading backwards. A few years ago, I found out that reading backwards any text is really informative. You probably know that postulate of transformative grammar that each statement in language has at least two layers of meaning: the deep structure, which contains a lot of information, and which we almost never use in current communication, and the superficial structure, which we use in the verbal expression properly spoken, and which is a contracted and generalised version of the corresponding deep structure.

I found out that when I read a text backwards word by word, which goes like ‘like goes which word by word backwards text a read I when that out found I’, and the text is really complex, that inverted reading sort of peels off layers of meaning and allows to get more information out of that piece. I am going to do the same, just idea by idea, and not really word by word, with that summary of business plan that I introduced in « Deux cerveaux, légèrement différents l’un de l’autre ».

So, the last idea, put in the first place, is that in the market of FinTech utilities, supposed to facilitate the development of new businesses, three pricing strategies are available. Strategy #1 is that of relatively high commission levied on each transaction done at the corresponding FinTech platform, and that relatively high price would be around 5% of the value being transacted. Strategy #2 goes the completely opposite way and sets the transactional fee at a level comparable to that applied by typical brokerage houses in the stock market, like 0,4% ÷ 0,7%. Strategy #3 tries to go sort of the middle way and sets the transactional fee at a rate similar to that charged on a typical bank loan for SME (Small and Medium-Sized Enterprises), which is currently around 1,6% if you know the right bank for you.

There is a catch in this strategic choice. There is a general assumption regarding the connection between financial markets and real business: the latter has a definite need for capital, based on the value of the current output, and on the parameters of the Cobb-Douglas production function. In this specific case, it is the aggregate need for capital from the part of small, local investment projects in the field of renewable energies, in Europe. Those guys need a definite amount of capital, let’s call it K, conformingly to the tradition of economic sciences. That stream of capital generates a stream of financial fees on giving that capital a lift on its journey, and that stream of financial fees is the market, where FinTechs dwell.

Strategy #1 defines a market worth 5%*K, Strategy #2 drives it down to (0,4% ÷ 0,7%)*K, and Strategy #3 settles for 1,6%*K. You understand? For each strategy, I assume that my EneFin project would be one among many to use it and thus each strategy would define a financial market. If the majority of players goes for Strategy #2, the really aggressive, low-price one, any FinTech business with higher fees, like in Strategy #1 or #3, would almost automatically face a glass ceiling. On the other hand, if the first FinTech settlers in that market go for Strategy #1, at 5%, it leaves to the others a whole range of possibilities. The choice between those three strategies is not only numerical. It is a choice between various levels of aggressive competition, too.

Whatever the final choice, the reverted reading of my own ideas has proven fruitful. I found a hole, to stuff up with information. How much capital do new projects in renewable energies really need, in Europe, at the aggregate level? I start that estimation with the data I have, and then I will do some rummaging online. I will use the theoretical construct known as LCOE, or ‘Levellized Cost Of Energy’. With It being the capital invested in period t, and Mt standing for the cost of maintenance in period t, accompanied by the cost of fuel Ft in period t, the real output of energy Et allows calculating LCOE with the formula below:

LCOE

With renewables, fuel is for free, hence Ft = 0. The capital that suppliers of energy need for investing in the new capacity of new projects, and for maintaining them in good condition is equal to K = ∆Et*LCOE. Quarterly reports published by the European Commission allow estimating the typical LCOE, in the market of renewables, at some €0,06/kWh, with a descending trend. In like 5 years from now, we can expect LCOE being closer to €0,04/kWh. As for the ∆Et, my own estimates allow ranging it between 82 TWh and 107 TWh a year. As I convert the terawatt hours into kilowatt hours, and do the ∆Et*LCOE, it gives €4,94 billion ≤ K ≤ €6,42 billion a year.

This is my own educated guessing. Now, I go and ask IRENA, i.e. the International Renewable Energy Agency. It proves to be a good idea. I find something interesting. According to my own estimations, the output of renewables in Europe, in 2017, was around 3 670,45 TWh, which, in turn, allows guessing an underlying capacity of 3 670,45 TWh/ 8760 hours = 419 TW. In means that what I computed as capacity actually used makes like 82% of the capacity installed. As I run the same kind of check for 2015, I have the output of renewables at 3 423,5 TWh, which corresponds to a capacity of 390,8 TW, and IRENA provides 465,4 TW of capacity installed, for the same year 2015. Capacity used divided by capacity installed, in this case, gives like 84%.

It generally holds in the presence of two-sided, anti-bullshit test. Capacity installed is usually greater that the one actually used. The opposite is impossible, whilst equality between these two is hardly conceivable, and risky for the energy system as a whole. If I base my calculations of capital needed on the delta of capacity installed, I need to add that overhang of capacity left in reserve, yet financed. I target this one as a surplus of 20%. Consequently, my calculation of capital required morphs into €5,93 billion ≤ K ≤ €7,71 billion a year.

I apply my three alternative strategies to that fork of aggregates, and I get the numbers shown in Table 1 below. It is clearly more than what I computed in « Deux cerveaux, légèrement différents l’un de l’autre ». Given the logic underlying these two alternative calculations, I would rather go for the second one, i.e. that in Table 1. There are more empirical anchors in this second calculation. Still, this is the estimated need for capital in the whole European market of renewable energies. The financing scheme proposed in the EneFin concept suits most of all those relatively small, strongly local projects. Probably, they make just a fraction of those aggregates in Table 1.

Table 1

Estimated value of the market in financial services for the suppliers of renewable energies in Europe, € billions
From To
Strategy #1  296,31  385,46
Strategy #2  29,63  38,55
Strategy #3  94,82

 123,35

As I see and feel it, that business plan for the EneFin project still needs some ripening. I am letting it crystallize in my brain. Take your time, business plan. Take your time. Anyway, what really counts as my personal creation is what will remain in this world after I am dead. I am changing my topic. I turn towards teaching and the preparation of syllabuses for the next academic year. I am using this same occasion to write some educational takeaways for my website.

As it comes to teaching, I feel like shaking it off a bit. Each year, I have more and more teaching hours, at the university. In the academic year which has just ended, it was over 500. In the one to come, it is likely to go over 700. I want to reassess my goals and my values in that process. This is an interesting thing I discovered quite recently, as a matter of fact: casual tasks can be done acceptably well without putting much of myself in the doing of them, but a big, solid workload requires real emotional involvement, at least if I want to do it really well and not to burn myself out in the process.

So, here is the list of the main subjects I have in my schedule for the next academic year:

  • Management: I teach it in two different, Undegraduate majors, and in two languages. I teach that curriculum for the major of International Relations, in English, and for the major of Film & Media Management, in English as well. Besides, I teach in Polish the same subject for the same major of Film & Media, Polish track.
  • Business Planning/Innovation planning – teach it in Polish, in the 3rd year of Management major for undergraduates. This is supposed to be an integrative, advanced course, with little theory and a lot of practical work.
  • Microeconomics – in the English track, I teach it in the first year of International Relations’ major, Undergraduate, basically to the same students whom I work with in the course of management; I am going to teach similar curriculum in the Graduate major of Management, English track as well; in the Polish track, this is a complementary course for the 1st year of Information Technology undergraduates.
  • Economic Policy – I teach this curriculum in the English track, 3rd year undergraduate in the major of International Relations.
  • International Trade/ International Economic Relations – the slash I put in the name of this subject means that it consists, in fact, in two similar subjects I teach at two different levels, i.e. Undergraduate (International Trade) and Graduate (International Economic Relations), of the English track in the major of International Relations
  • Political Systems – English track, Graduate level, major Management

Good, so this is the inventory of what I do, and now my big question is: what do I really want to do in what I do? What do I want to achieve in terms of teaching outcome? What do I want my students to learn?

The first part of my answer is pretty simple: I want them to learn science as such, i.e. the scientific method, and I want them to become basically proficient in using that method for building their own social roles in an intelligent way. I want to train them at the sequence made of observation followed by hypothesising, which is further followed by empirical verification and communication of results. One of the big lessons you can learn, when practicing this sequence, is that it takes time and smart coordination of your own actions. Science, understood as a method of dealing smartly with that bloody mess we call reality, requires both creativity and rigour, and these two traits are the second major takeaway I want my students to have, in my classes.

This basic education goal comes from two plain observations. Firstly, there are things you can learn in a classroom, and those which you just can’t. Secondly, we are coexisting with an emergent form of intelligence, the digital one, and we need to adapt. Can you learn management in a classroom? Certainly not. Management is a craft and an art, in the first place, and only secondarily this is a science. As craft and art, management is, most of all, fierce inter-personal competition inside a complex social structure. Yes, there is ethical management. Yes, there is management for higher a purpose. Yes, you can imagine ethereal beings like that. Still, if you want to do any management at all, you have to stay in the game inside an organization, and in order to achieve that simple goal, you need to be efficiently competitive.

Can you learn international relations in a classroom? Once again, the answer is: hell no. If, one day, you try to put together a policy, a social action, or a business at the international scale, you can really learn what international relations are. If you can’t learn management in a classroom, and can’t learn international relations, what is the point of teaching them at a university? If you ask this question, you are bloody touching the essence of the thing, as I feel it. I named some of the things you can’t learn in a classroom, so what are the things you can?

In order to understand the learning you can have out of any situation, the essential thing is to understand what is happening in that situation. In a classroom, we are basically having a conversation in a semi-organized group. Semi-organized means there is a basic division of roles – I teach, students learn and take exams – but we have a lot of flexibility when it comes to ritualizing the whole situation. I can play the distant, cold, exigent bastard, or I can play the nice guy. Students can play the smart ones, the diligent ones, or can play fools.

If you have a semi-defined social role, both you and your social environment will push towards defining it completely. This is what we, humans, do: we shape our social structures so as to make social relations acceptably predictable. Thus, in a classroom, the most fundamental process of learning that takes place is the transition from semi-defined social roles to the fully defined ones. Students essentially learn how to make themselves into the social role they think their social environment expects them to endorse.

I remember one student, from the last Winter semester. He had that special style, combining Gothic elements, such as demon-looking, all black contact lenses, with a vaguely androgynous clothing and haircut. Apparently, a rebel. Still, it was nothing short of amazing to watch this alleged rebel in his virtually desperate efforts to find a role in the group, i.e. in relations with other students and with me. That guy was literally experimenting with himself, and tried to play a slightly different role at each of the 30 classes scheduled in the semester.

Thus, in a classroom, what we have in terms of the most fundamental teaching, is a partly controlled environment for training students in the definition of their own social roles. It is like: ‘Look, you can read this book, and you can use that reading to endorse various stances. You can play the dumb one, who doesn’t understand anything. You can play the diligent one, who wants to get extra points by writing an optional essay. You can play the smart guy, and challenge me in class on the grounds of that book. In short, you can learn how to use large parcels of theoretical knowledge in order to experiment with your own social identity’.

OK, enough writing in that update. I feel like switching, as it is my ritual, to French, once again, to make my brain dance to a slightly different linguistic tune.

I am consistently delivering good, almost new science to my readers, and love doing it, and I am working on crowdfunding this activity of mine. As we talk business plans, I remind you that you can download, from the library of my blog, the business plan I prepared for my semi-scientific project Befund  (and you can access the French version as well). You can also get a free e-copy of my book ‘Capitalism and Political Power’ You can support my research by donating directly, any amount you consider appropriate, to my PayPal account. You can also consider going to my Patreon page and become my patron. If you decide so, I will be grateful for suggesting me two things that Patreon suggests me to suggest you. Firstly, what kind of reward would you expect in exchange of supporting me? Secondly, what kind of phases would you like to see in the development of my research, and of the corresponding educational tools?

Support this blog

€10.00

 

When is it the right moment to expose ourselves?

My editorial

And so I know what I want. I mean, I sort of know what I want regarding some aspects of my life. That business I want to put together in connection to smart cities is a centre of experimental research focused on studying human behaviour in the interaction with smart technologies. The business idea behind is to supply experimental knowledge, and the tools for discovering new experimental knowledge, to businesses which would like to accelerate their innovation and to smoothen their relations with big institutional customers. The metric of utility I have come up with is the possible reduction of payment lag in those customer relations. This is what I have come up with so far, and you can find it in my last update in French: Ça semble tenir le coup. Ça promet .

The business concept I am entertaining for this experimental centre is based on sort of an educated guess, which I have formulated, over years, as I have been studying different business cases: when something lasts longer than it technically should last, someone in that process is learning something by trial and error. When an IT company, like Asseco Polska, whose financials I highlighted in that last update in French, used to receive payment for its services after 80 days, on average, in 2011, and this time extended to 101 days in 2016, the process of interaction with the average customer must have changed. Of course, some of that extension is due to changes in the financial liquidity in customers themselves. Still, as I try to reconstruct the interaction between an IT provider and an institutional customer, there is that component of tailoring the solutions supplied to the exact needs and characteristics of the latter. The supplier learns by doing, and that learning frequently manifests itself as an additional lag in payment: the customer withholds a part or the total of the sum due to the supplier until the latter removes from the solution provided what the customer sees as flawed.

This phase of a contract is quite tricky. It is crucial to separate true necessity of improvement in the solutions provided, on the one hand, from tactical withholding of payment, on the other hand. Said necessity of improvement seems to follow a common pattern, well-known from the management literature: engineers think in terms of blueprint (coding in the case of programmers), whilst the end-users of a technology think in terms of the most basic utility. That basic utility has to include fool-proof safeguards: if clicking the wrong link on the screen blocks an entire accounting system, there is something wrong.

You would say ‘No, these things don’t happen anymore. Digital solutions are past that phase’. Well, recently, I attended a presentation, by a fellow scientist, about how he and his team had improved some utilities in the SAP system. For the mildly initiated: SAP Business One, marketed by SAP SE, Germany, is one of the most advanced ERP-class systems in the world, maybe even the most advanced one. So, those fellow scientists from Krakow (Poland) did a simple thing: they took some parts of the SAP interface and they submitted it to behavioural testing, where users were being observed with an eye-tracking device. After the tests, the interface has been modified so as to optimize its functionality for the eye-movements of the users. Result: a series of operations which took the users 6 minutes before the modification, dropped to 1,5 minutes. Yes, this is a fourfold rise in productivity, and it was just one tiny piece of the total functionality available in SAP Business One.

My home university, the Andrzej Frycz Modrzewski Krakow University (Krakow, Poland), prides itself to rely on 100% their own technology. Library, grading, syllabuses, e learning platform: all that is genuinely made for the university by an appointed team of IT engineers, without any Google-made prosthesis. Still, man, sometimes you wish it was Google-powered somehow. It takes years to optimize each piece of integrated software so as to make it really functional. A lot of this optimizing relies on communication, which, of course, is what communication usually is: imperfect. I imagine the same thing happens in hundreds of cases, when an integrated technology (it does not have to be exclusively digital, you know) is being implemented in a large social structure. It is only after starting the implementation that engineers face the real users’ behaviour, and this is when the real tailoring starts.

Summing it up, partly, here is my idea: providers of complex technologies, when they have a big institutional client, go through that tricky phase in the contract, when the technology is basically in place, the invoice has been issued on the buyer, but the latter is still no satisfied and withholds the payment, and his dissatisfaction is well-founded, as the true utility of the technology installed is only taking shape, and it brings some surprises. The experimental centre I want to create would run experiments useful in minimizing the trial-and-error component in such situations. I would like to create a body of experimental knowledge about human behaviour in interaction with smart technologies, and experimental tools for developing more of such knowledge, so as to shorten the payment lag that the suppliers of smart technologies experience in their relations with big institutional customers.

Now, I am advancing a behavioural hypothesis: any large social structure, when absorbing a new technology, behaves like a mean institutional customer, and it pays the full bill only after it is fully satisfied with said technology. For those of you who want solid theory, here comes a piece: Robertson, T. S. (1967). The process of innovation and the diffusion of innovation. The Journal of Marketing, 14-19. The idea is the following: when you are a supplier of advanced technologies and you are selling your solutions on kind of a retail basis, to many small customers, they all make sort of a community in the sense that their behaviour regarding your technology is coordinated through socially recurrent, partly ritualized patterns. In that community, you will find the early innovators, the early adopters, the sceptical late adopters, and the fence-sitters. You can fully cash your returns on investment only when all those customers have absorbed your technology. If you are launching your technology in a form adapted to the tastes of the early innovators, it is going to bump against the wall of different tastes observable in more conservative customers. You need to modify your technology so as to make it easier to swallow conservatively. Modifications take time, and you need much of that time just in order to figure out the right modifications to make.

I think I can start generalising those ideas in the form of a model, regarding the interaction between the suppliers of technologies and their customers. Thus, the mission of my experimental centre will be to reduce the cost of innovation, or CI, incurred by the suppliers of smart technologies for smart cities. The cost of innovation is made of two component costs, namely: the cost CIN of ongoing invention in the strictly spoken new technologies (i.e. those not marketed yet), and the cost CIM of improvement/modification in the already marketed technologies. Both CIN and CIM contain two further parts: the cost of engineering work strictly spoken (blueprinting the technology), which I am going to symbolize with ‘B’ in subscript, and the cost of discovering behavioural patterns in the end-users of the technology in question, and I symbolize this one with ‘D’ in subscript.

Right, I am an economist, and so I need writing some formulae, so here is what I am thinking in mathematical terms:

[CI = CIN + CIM = CIN;B + CIN;D + CIM;B + CIM;D] =>  [CI = CIB + CID]

The last part of that complex logical implication, or ‘=>  [CI = CIB + CID]’ means that with the assumptions given above, the total cost of innovation is, in other words, the sum of what we spend on the strictly spoken blueprinting (CIB), on the one hand, and the expense on patterning the users’ behaviour (CID).

The mission of my experimental centre will be to shave off the CID. Next question: how the hell do I want to achieve that? Next piece of theory coming, then. The process of innovation, i.e. the process of blueprinting a technology, and optimizing it as user-friendly, can be represented as a series of interactions involving a prototyped technology, and a user testing it. Each prototype is an educated guess about the optimal utility for the users. Each interaction of a prototype with the users who are testing it can be a success (correct guess) or a failure (go and figure it out again, anew). In the process of innovation, we have n interactions, with p successes and q = n – p failures. The total CID cost comprises both the expenses connected to successful trials, and to the failed ones. Now, I start sailing uncharted waters, at least uncharted for me, as I am asking myself: how can I reduce the total cost of that process and/or increase its efficiency?

In my previous update in English, the one entitled ‘And so I ventured myself into the realm of what people think they can do’, I started developing on that idea that p successes and q = n – p failures can happen as many distinct sequences, technically equiprobable, but actually very different in their outcomes. My most personal intuition, which comes from my academic teaching, my sport practice, and my experience with the Wim Hof’s method of breathing, all that tells me that the sequence which works the best for learning is the following: ‘serious ass-kicking, technically a failure >>> small success >>> small success … (a whole series of small successes) >>> another ass-kicking (another failure) >>> and I loop on a series of small successes’ etc. My nervous system learns new skills through a sequence of successful attempts, not through a sequence of failures. Failures just teach me what not to do, but I need those small successes in order learn what to do, or form the right pattern of doing things. Still, failures keep me sharp and alert, they prevent me from becoming too self-enclosed, or, I should rather say, failures can keep me in that blessed frame of mind under the condition of being properly taken in.

Applying this general wisdom to experimenting with new technologies implies that I can calibrate successes and failures into different sizes, sort of “big success <> small success”, “big failure <> small failure”.  I imagine an experimental sequence, where engineers start with confronting an advanced prototype with testing from the part of users, and I arrange this particular step so as to make failure as probable and as deep as possible. I select the testing users, and I design the conditions of testing so as to put every possible hurdle and trap on this path. I want the engineers to bounce against something they are very likely to perceive as a brick wall. This is the initial ass-kicking. The next step consists in studying that failure very minutely, so as to own it completely. This phase of the experimental sequence brings two major learnings. Firstly, it shows what happens to our technology in a really unfriendly environment. This is the equivalent of those IKEA armchairs being crushed by a mechanical press thousands of times a day. Secondly, it teaches the engineers how they behave, as human beings, in the presence of a major failure. Behavioural study of the engineers is as important at this stage as that of the users.

In Stage 2, engineers modify the prototype on the grounds of the ass-kicking received in Stage 1. This time, it is important to break down the sequence ‘modification >> testing’ in very small steps, so as to make successful guessing of the users’ needs as probable as possible. In this stage, engineers learn to interact with users on a current basis, and, hopefully, create a series of successful – i.e. fully functional from the users’ point of view – modifications.

Now, the big question is: how to alternate Stage 1 and Stage 2. In other words, when is it the right moment to expose ourselves to the functional outcomes of every possible flaw in our technology? I have to ruminate it a bit.

I am consistently delivering good, almost new science to my readers, and love doing it, and I am working on crowdfunding this activity of mine. You can consider going to my Patreon page and become my patron. If you decide so, I will be grateful for suggesting me two things that Patreon suggests me to suggest you. Firstly, what kind of reward would you expect in exchange of supporting me? Secondly, what kind of phases would you like to see in the development of my research, and of the corresponding educational tools?

There are many ways of having fun with that experiment

My editorial

I am designing an experimental environment for products based on digital technologies. I am coalescing my thoughts around the type of digital products that I am particularly focused on those last weeks, namely on the FinTech. I started sketching the contours of that experimental environment in my last update in French (see Une boucle de rétroaction qui reviendra relativement pas cher ). Now, I am rephrasing the idea in English and giving it a bit of an extra spin. There are two groups, the geeks and the ordinaries, I mean a group of k software engineers – or k teams of them, the unit is to settle later – and a group of n users, allegedly prone to being more or less enthusiastic about FinTech products. The first round of the experiment starts as the users are asked to make a set of financial decisions in a controlled environment. By controlled environment I mean a standardized budget, e.g. €10 000, and a standardized set of possible uses for that small fortune. It would be good to have pretty a wide range, as uses come, covering typical saving (holding liquid capital for future, unspecified uses), investment (buying future possible gains with the present allocation of liquid capital), and typical consumption (e.g. sex, drugs, and rock’n roll).

The users make their financial decisions in that controlled environment. The software engineers are then being presented with various types of data about those decisions. ‘Various’ can range from simple reports like ‘50% of the users decided to invest not less than 40% of their budget into the Smartest President’s next business’, through more and more elaborate data on individual decisions, taking a slight turn by a detailed database of those individual decisions, all the way to the data from behavioural observation of the users (i.e. sequence of clicks, sequencing of the decision-making process, speed and pace of the corresponding sequence, eye-tracking, pulse, breathing, don’t-put-that-thermometer-where-we-can-both-regret-it etc.). Yes, you guessed correctly: we have been just experimenting with the users, and now we are experimenting with the software engineers. What type and what amount of data will be actually useful for them? If they can choose the type and the scope of information provided about the users, what their choice is going to be? From then on, what their response is going to be? ‘Cause now, the software engineers are to present prototypes of FinTech products, as well adapted as possible to what they know about the users.

Here’s the little caveat: the users are not supposed to tell the engineers what they want. There is just objectified observation. Why? Well, what I want is to simulate, in that experimental environment, the real working of a market, only faster. In real markets of standardized digital products, engineers seldom speak in person to the end users. They have reports, and they base their invention on it. In my experimental environment, each engineer can pick and choose the parcel of information for takeaway, and then calibrate their engineering response accordingly.

In that first round of the experiment, I arrange an interaction between the users of FinTech products and the engineers supposed to create those products. At this stage, the experiment is already informative at many levels. At the first and most basic level, it tells is about the patterns of financial decisions observable in users, and about the way they make those decisions. One level higher, I encompass both the data on users and the data on the engineering response, and what I get is knowledge about the interactive engineering process itself. In the presence of the given set of data on users, their decisions, and their decision making, how quickly did the engineers respond by producing prototypes of FinTech products? What is the typical pattern of utility in those products, and what are the idiosyncrasies in individually created FinTech solutions? I go one level higher in the generality of observation, and I am connecting the type and scope of data on users with the pattern of engineering response. Did richer a set of information on the users’ behaviour contribute to create more sophisticated FinTech products, i.e. did those engineers, who chose to work with more extensive an information about users, create something clearly sticking out of the crowd as FinTech solutions come? Maybe there is a golden path of wisdom, as regards the amount of data on users, somewhere between a cursory report about what percentage of them chose to buy government bonds, and extensive a report on how was their body temperature changing during the decision-making process.

Besides the amount of data collected about the users’ behaviour, there are other dimensions in that experiment. The specific profiling of users is, of course, one of them. How does the whole process work with users of different age, economic status, sex, education, professional walk of life and whatnot? Another dimension is the relative complexity of the users’ behaviour, on the one hand, and the complexity in engineering response, on the other hand. Up to a point, the more people we have having a certain type of idea, the more distinct ideas are being crystallized. In a sample of 100 users we can probably come up with greater a number of distinct patterns in the making of financial decisions than in a sample of 10 users. Still, diversity in users is not just their sheer number, it is also the relative disparity of their characteristics. If, in a sample of users, we put people with high school degree, and those with their college diploma hung on the wall, we come with a certain diversity in decision-making. Now, we add those with Master’s degrees. What has changed? Did the addition of one more level in education change the diversity in the making of financial decisions? Now, we can experiment at the same level with other characteristics, and each time the question is the same: how does the relative complexity in the population of users reflect in the relative diversity of the observable patterns in their financial decisions, and in the process of making those decisions?

Just to return to real life, which is sometimes useful in science, I remind: in any type of business, the supplier has to deal with various degrees of complexity in their target market. I am a software company from Romania, I had come up with a super-cool payment platform in the Romanian market, and now I am trying to see the big picture, i.e. the Chinese market. I will be interacting with much bigger, and potentially more diverse a population of customers. How should I change my engineering processes in order to cope efficiently with the situation? This is precisely the kind of questions that my experimental environment can give an answer to.

We go one step further, and we can test the link between the above-mentioned complexity in the set of users’ characteristics, and the complexity of the solutions prototyped by engineers. If I add one more social stratum to the sample of users, does a constant population of engineers come up with more of distinct prototypes of FinTech products? There is another path going (somewhere) from this point: if I keep the complexity observable in users constant, and I change the diversity in engineers, how will it affect the creation of prototypes? In the presence of a constant sample of 100 users, with their descriptive data, will 50 engineers, each working independently, generate greater a diversity in prototypes than 10 engineers? Once again, is there any golden path of optimization in that proportion? What if, besides varying the number of engineers, I shift between various types of diversity in their characteristics, e.g. the number of years spent in the professional engineering of software?

In all that experimenting, and shifting diversities, an economic equilibrium is timidly raising its head. What does an economic equilibrium’s head look like? Well, you can take the Marshallian equilibrium as a good example: two convex curves, crossing at one point, so it is like a Guinea pig with a pair of pointy ears, pointing slightly outwards. This could be something like a rabbit, as well. Anyway, each given set of characteristics in the sample of users, combined with a set of characteristics in the sample of engineers, produces an outcome in the form of a set of prototypes in FinTech products. A set of FinTech products, in turn, generates a certain stream of financial gains for users – like the money they save on paying for electricity or the money they gain from an investment portfolio – and a stream of revenue for the supplier of FinTech. The latter earns money in two basic patterns: either by collecting a small margin on each transaction performed by the user, or by charging the user a fixed monthly fee for the access to the given utility. I suppose that a well-rounded prototype, in a FinTech utility, should contain those components.

Thus, in my experiment, I have the following, compound variables, modifiable in my experimental environment:

  1. a vector UR = {ur1, ur2, …, urm} of m characteristics observable in users;
  2. a vector EN = {en1, en2, …, enk} of k characteristics observable in engineers;
  3. a vector EN(UR) = {en(ur)1, en(ur)2,…, en(ur)l} of l users’ characteristics actually used by the engineers in their work; in the experimental environment this vector can be either exogenously constrained or freely shaped by the engineers themselves;
  4. a vector FU(UR) = {fu(ur)1, fu(ur)2, …, fu(ur)p} of p functional utilities available in those FinTech prototypes; you know, the number of clicks (or screen touches) you need to transfer all your money to the Caiman Islands etc.;
  5. an aggregate stream PF(UR) of users’ payoffs from those FinTech prototypes presented; it can be further decomposed as a vector, namely PF(UR) = {pf(ur)1, pf(ur)2, …, pf(ur)z} of z users’ payoffs, but in my basic scheme I want to use it as an aggregate stream;
  6. an aggregate stream P*Q of revenues that the suppliers of those prototyped FinTech products can derive from their exploitation; once again, P*Q can be also represented as a vector P*Q(UR) = {p*q(ur)1, p*q(ur)2, …, p*q(ur)ß} of ß specific streams per user or per utility, but in my basic model I want to keep it simple;
  7. an aggregate stream EC of engineering and maintenance costs, in suppliers, connected to the basket of FinTech prototypes proposed; again, translation into a vector is possible, although not always necessary;

Now, my basic economic equilibrium, if I keep it more or less Marshallian (you know, the rabbit), would be something like: [P*Q – EC] -> PF(UR). It means that there could be some combination of my experimental variables, which makes the aggregate profits, on the part of suppliers, tend to equality with the aggregate payoffs in users. That Marshallian approach has some catches in it. It supposes the existence of an equilibrium price in FinTech products, i.e. a price that clears inventories and leaves no customer with unsatisfied demand for FinTech utilities. This is tricky. On the one hand, in finance, we assume the existence of at least those most fundamental equilibrium prices, like the interest rate or the discount rate. It was observed already in the 18th century, by Adam Smith, among others, but, interestingly enough, not earlier. When I read that book by Jacques Savary, published in 1675, entitled ‘Le Parfait Négociant ou Instruction Générale Pour Ce Qui Regarde Le Commerce’ (in English: ‘The Perfect Merchant or General Instructions as Regards Commerce’), equilibrium rates in finance tend to be considered as a comfortable exception than as the general rule of the market. Still, here, in FinTech, we are at the junction of finance and digital technologies. In IT, people don’t really like talking about equilibrium prices, and I can hardly blame them, ‘cause the actual diversity in IT utilities is so great that we can throw the assumption of homogenous supply through the (metaphorical) window as well.

Anyway, I can optimize my experimental environment regarding the Marshallian equilibrium, or any other economic equilibrium, actually. I can make a function [P*Q – EC] = f[UR], i.e. a function linking the aggregate profits in the suppliers of FinTech products with the observable complexity in the population of users. I can do the same with the complexity observable in the population of engineers. There are many ways of having fun with that experiment, at this stage, and this is just the first stage. There is more to come. The engineers come up with a range of prototypes in FinTech, and they present them for testing. The users test those prototypes, in an experimental environment, and once again we apply the same philosophy: the users are supposed to be informative by their behaviour, not so much to talk about their impressions.

An interesting question arises, namely that of learning. If I am one of those users, and I am supposed to test those prototypes devilishly, I mean like 6 prototypes tested over 6 days, with 6 cups of coffee in terms of neuro-logistic support, I can’t help learning. At my 3rd prototype tested, I will be using, consciously or not, whatever, the experience accumulated with the testing of prototypes 1 and 2. Thus, what I will be really testing, will not be each prototype separately, but a sequence of prototypes, associated with a process of learning. Someone could say that learning creates a harmful bias (believe me, there are people who think so), and so the edge of learning can be blunted in two basic ways.

Firstly, the experimental environment can take away from me any feedback on the financial decisions I make with a given FinTech utility. When we do something about money, we like seeing the outcomes, it is kind of attached to money in its cultural code. You take away the feedback on results, and, by the same means, you take away the incentive to learn. We could think about such an experimental environment, and yet, I have like a little voice in the back of my head (no, I am perfectly sane) saying that financial decisions without feedback are not really financial decisions. That would be an experiment biased by the absence of bias. Secondly, I can make each user test just one prototype. There is no opportunity for learning in this case. Still, we need to split the sample of users into as many subsamples as we have prototypes to test, and each has to somehow representative.

Now, do I really want to take learning out of the equation? Do I expect the users out there, in the big, wild market of FinTech, using those apps without learning anything? I just think it could be safer to assume that people learn things. What we should be really testing, at this stage of the experiment, could be precisely the phenomenon of learning, with respect to FinTech.

I am working on making science fun and fruitful, and I intend to make it a business of mine. I am doing by best to stay consistent in documenting my research in a hopefully interesting form. Right now, I am at the stage of crowdfunding. You can consider going to my Patreon page and become my patron. If you decide so, I will be grateful for suggesting me two things that Patreon suggests me to suggest you. Firstly, what kind of reward would you expect in exchange of supporting me? Secondly, what kind of phases would you like to see in the development of my research, and of the corresponding educational tools?