Philippe's Official Website

Future Skills in the AI and post-AI era

2024-03-17T17:55:00Z

Today, with Wikipedia and Google in our pockets, we no longer need to remember all historical details, but we need to have an opinion and the willingness to learn. Every movie-maker, every game designer, every author seems to have a political motivation and sitting back and enjoying is no longer an option. After watching a recent movie about Cleopatra, Napoleon, Vikings, or the Roman Emperors one should have the reflex to verify facts. For example, one will notice that each of those movies require a voluminous book to enumerate inaccuracies. Even more important is to understand why the movie-maker did history such injustice. The lesson is that humans are inferior to machines when it comes to remembering facts. This has been established a while ago, and education is already adapting to it. However, there is much more to come.

With the benefit of hindsight, all changes seem obvious. However, it is nearly impossible to predict inventions and changes that we yet have to see. For example, since the 1950s a career in IT and programming has been guaranteed to be a great choice. With the advent of AI software engineers (like Devin, last week) that will soon be over. Till 2 years ago, the best advice one could give a student was to study and be ambitious in the pursuit of the highest intellectual challenges. That is no longer a silver bullet. It appears that intellectual jobs are easier to be replaced by machines than jobs that require navigating the complexities of the physical world.

We are at a turning point of history, and just as in the 1910s, we had no idea what the car or plane would do to our landscape, habits, health, cities, warfare, etc. Today we have only faint clues what AI could do to the fabric of our society, but we have no crystal ball that tells the future.

While we believed that we would be superior in thinking, strategy (eg. chess and go), strategy with imperfect information (e.g Stratego), creativity, reading emotions, guessing intend, etc. All of this became history in the last decade. Robots are better in strategy games, are already fantastic artists, creators, and they are genuinely creative, etc. On top of this, machines are able to digest more information in minutes than any person could do in a whole life.

People that will still be needed are:

empathic leaders: with the Doomsday Clock at 90 seconds before midnight the world needs more than ever people with an open mind, people who want to do good and make the ethical choices even when no-one is watching, people who want humanity to succeed, people who can bring diverse individuals together;
physical workers: people finishing the build of a house, working on high voltage cables, carers and nurses. These jobs are hard to replace by machines and humans will need to do them for a while.
scientists to prepare the next industrial revolution beyond AI. Periods of economic growth seem to be a good way to prevent destructive wars (e.g. both world wars happened in periods of economic depression). After AI, we will see the revolution of quantum computers, and after that I’m rooting for nanotechnology and biotechnology. While AI will help dramatically in those endeavors, the world needs scientists to prepare those future revolutions. The scientific and technological challenges have been increasing throughout human history, and we will need AI to help. However, only the scientific method has brought progress, and even if we could, we should not leave this up to AI alone;
original thinkers who value virtue, critical thinkers who can see through a world of deep fakes. In a world where everything could be fake, it becomes easy to be misled. The world needs more people who act intentionally virtuously, especially when no one is watching;
people who value the physical world, soon it will be a delightful alternative to hide in a fantasy world of virtual reality; it will most likely become an addiction similar to narcotics causing people to become dysfunctional citizens. This addiction seems much more dangerous than alcohol or narcotics. Indulging in excessive gaming, VR, or immersive movies is legal and easy to start and much more than narcotics one can exactly choose to be what one wants. On top of that, VR will soon be nearly indistinguishable from reality, and hence an excellent escape the complexity and frustration of their physical existence;
people with professions that we don’t have now. Till a few months ago, the job “prompt engineer” did not exist. Thousands of great professional choices that exist today didn’t exist 20 years ago or have been transformed dramatically. Since it is not possible today to predict what will be a good job in the next decade, it is essential to be flexible, open-minded, and continue to learn and adapt. What we need is lifelong learners.

Skills and professions that will disapear

It would be equally valuable to think of professions that won’t be needed anymore or will profoundly change due to the mass adoption of AI. Jobs such as medical doctors, judges, artists of all kinds, programmer, designer, architect, driver, food delivery, etc. will still be important, but AI will dramatically change how it will be done. The spectrum of change will be from total annihilation of the human in the role (soon to happen for e.g. food delivery, personal trainer, etc.) to AI assisted human. Even for those jobs where humans make the ultimate decisions, it is plausible to think that a model of mass solutions executed by robots and humans considering the outliers – think along the lines of the 80/20 rule. For example 80% of medical diagnoses can be done by a robot (e.g. diagnosis of a common cold), and only 20% of patients consult a doctor. While today, most people value the human touch (for a driver, personal trainer, or doctor for example) soon enough AI will outperform its human counterparts so much that the human alternative will be seen as inferior. No human doctor, for example, can keep pace with the vast amount of research being produced, monitor constantly parameters such as weight, and hundreds of molecules in urine and excrement as your personal medical assistant connected to sensors your toilet can do every day. Within decades humans will be far left behind in the precision of their diagnosis, quality of follow up, and personalization of the service.

Keeping this in mind, I would argue that the following skills will be a good investment:

Critical Thinking and Problem-Solving. The ability to approach problems from different angles, think critically about possible solutions, and solve them effectively is crucial. Using the scientific method to distinguish fake from fact.
Digital Literacy and Coding. While the number of programmers will be more limited, it will be critical to audit machine generated code, we need a lot more understanding to keep machines benevolent and transparent, and we need to keep making progress.
Data Science. As AI and machine learning become more ubiquitous, understanding how to interpret and work with data remains a valuable skill. This also ties in with understanding what conclusions are true and which not. Furthermore, while bias in data and models can be addressed there is no silver bullet and ethical choices become more important (e.g. the famous question of who the AI driver should kill in case killing cannot be avoided, or should we –in order to avoid gender discrimination– not use gender for car insurance and hence increase prices for all? Or should ladies finance the damage done by men that driver more reckless?)
Emotional Intelligence and Communication. The ability to understand and manage emotions, and communicate effectively will remain vital. AI can help here a lot, but it will remain important to be able to speak up in a meeting and formulate a coherent opinion. Sure, we can have AI that puts ideas in our mind or on our AI eye lenses, but that is unlikely to be our opinion
Resilience, Motivation, Leadership, Social Influence, and Technological Literacy. While an AI manager will soon be possible, in inspiring leader is bound to be human for long decades.
Adaptability and Lifelong Learning: The rapid pace of technological change has brought us to a volatile, uncertain, complex and ambiguous world (aka “VUCA world”), this means that adaptability and commitment to lifelong learning are essential.
Integrity, self-awareness, courage, and a deep understanding of ethics. To learn from mistakes and improve, people who own their mistakes, people who understand that their triggers and weaknesses are their responsibility, people who need no safe place to hide from facts and opinions, people who own their thoughts.
People who accept their fate and focus on what is within their control. People who show resilience in the face of adversary, take responsibility over own thoughts and actions, don’t look for excuses or an easy way out, but have the courage to play the cards they have been dealt.

Conclusion

In conclusion, the AI and post-AI era will demand a diverse set of skills. As we navigate this rapidly evolving landscape, critical thinking and problem-solving will be paramount, enabling us to discern fact from fiction and approach challenges from various perspectives. Digital literacy and coding will remain important, not just for creating new technologies, but for understanding and auditing the ones that already exist. Data science skills will be invaluable as we strive to make sense of the vast amounts of information generated by AI systems. In order for humans to remain in the driving seat, school should focus on production passionate people that follow the ethos of a virtuous life and can defend an ethical stance.

Create Your First Covid-19 Related App with R and Shiny

2021-05-14T19:13:00Z

The library {shiny} provides the tools to build interactive html-apps from your data project. It is the ideal way for users to experiment with the data and gain deeper insights than what one can expect from a static report.

You might know my Covid-19 Dashboard. It was one of the early dashboards build in March 2020 and remains one of the only dashboards that allows you to build your own models, visualise data in many ways, and draw your own conclusions.

It is a little too ambitious to explain all details in this dahsboard in this tuturial, but we will get you started with the basics.

While it is not strictly necessary to use RStudio, it will soften the learning curve and make your more productive.

In RStudio, select:

File New (or use the shortcut )
Select Shiny Web App ...
Fill out the form: select a name for your app, leave the single file option and select a directory where to save the app.
Click Create.

That is all: you are now presented with the code of the app in RStudio. You can execute it by clicking runapp: . The application that we have now is fully functioning, but does not contain the content that we desire.

The mains structure is as follows:


library(shiny)
ui <- fluidPage(
   # The presentation (html) goes here and
   # input variables are created here.
)

server <- function(input, output) {
    # Server logic: output variables are defined here
}

# This line runs the app, using the ui and the server
shinyApp(ui = ui, server = server)

In the user interface, it is the function fluidPage() that sets the general presentation aspects. This particular function will use the css from Twitter’s boostrap framework. The arguements passed to that function generate the html. Usually the names of the functions are self explanatory (e.g. titlePanel() generates a tittle panel with the string variable that it takes as argument).

The function sliderInput() will produce a slider that allows the user to choose a number. Its first argument is a string variable (in this case “bins”). This will make the variable input$bins available in the server function, which allows us to access the number that is chosen by the user.

The plot generated by the server function, can be accessed via special purpose functions. In our case the plot can be retrieved via the function plotOutput(). That function takes one string argument that is the same name as the output variable generated by the server function.

In the server function, we use the input variables to generate the output variables (e.g. output$distPlot)

Now that we understand the basics, we can start modifying the skeleton app to suit our needs. We will need

to connect to our data
change the ui
change the server function

To connect to live (or at least last night’s update), we will connect to the data. In order to keep the code in the app clean and readable we will move the functions for this to a new file global.R. This file is best placed in the same directory as the file app.R

We will download a copy of the data that resides on [githubusercontent.com], immediately load all libraries that we need and add functionality to clean up the data.


library(tidyverse)
library(plotly)
library(lubridate)
library(flexdashboard)

get_data <- function() {
  lnk_conf <- 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_confirmed_global.csv'
  lnk_death <- 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_deaths_global.csv'
  lnk_rec <- 'https://raw.githubusercontent.com/CSSEGISandData/COVID-19/master/csse_covid_19_data/csse_covid_19_time_series/time_series_covid19_recovered_global.csv'

  copy_data <- function (theLink, theLabel) {
    d1 <- read_csv(theLink)
    d1 <- d1 %>% gather(key=`Date`,value=theLabel,-"Province/State", -"Country/Region", -"Lat",  -"Long")
    d1$Date <- ymd(as.Date(d1$Date, format="%m/%d/%Y"))
    d1$`Province/State` <- ifelse(is.na(d1$`Province/State`), d1$`Country/Region`, d1$`Province/State`) # replace empty state names by country names
    d1[complete.cases(d1),]
    colnames(d1) <- c('Province', 'Country', 'Lat', 'Long', 'Date', theLabel)
    d1
  }
  d_conf <- copy_data(lnk_conf,  'Confirmed')
  d_recv <- copy_data(lnk_rec,   'Recovered')
  d_deat <- copy_data(lnk_death, 'Deaths')
  #max(d_conf$Date)

  d1 <- left_join(d_conf, d_recv, by =  c('Province', 'Country', 'Lat', 'Long', 'Date'))
  d1 <- left_join(d1, d_deat, by =  c('Province', 'Country', 'Lat', 'Long', 'Date'))
  d1 %>% replace_na(list(Confirmed = 0, Recovered = 0, Deaths = 0))  %>%
    mutate(Sick = Confirmed - Recovered - Deaths)
}

# takes the data from get_date() and eliminates the Province/State thing
get_dts_countries <- function(d) {
  d1 <- d %>%
    group_by(Date,Country) %>%
    summarise(sum(Confirmed), sum(Recovered),sum(Deaths), sum(Sick))
  colnames(d1) <- c('Date', 'Country', 'Confirmed', 'Recovered', 'Deaths', 'Sick')
  as.data.frame(d1)  # to make the grouping permanent
}

# Get the data timeset for one country
get_dts_country <- function(d, theCountry = 'ALL') {
  if (theCountry == 'ALL') {
    d1 <- d %>%
      group_by(Date) %>%
      summarise(sum(Confirmed, na.rm = TRUE), sum(Recovered, na.rm = TRUE),sum(Deaths, na.rm = TRUE), sum(Sick, na.rm = TRUE))
    colnames(d1) <- c('Date', 'Confirmed', 'Recovered', 'Deaths', 'Sick')
  } else {
    d1 <- d %>% filter(Country == theCountry)
  }
  as.data.frame(d1) # to make the grouping permanent
}

# get_last_numbers
# returns the last value of the time series
get_last_numbers <- function(d, theCountry = 'ALL') {
  d1 <- get_dts_country(d = d, theCountry = theCountry)
  # remove date and country name so we can use arithmemtic on the whole vector:
  if (theCountry != 'ALL') d1 <- d1[,-2]
  as.vector(d1[d1$Date == max(d1$Date),])  # data at last mindnight
}

Later, we will add more functionality in global.R, but we will first make sure that it loads properly. To do that we simply place the line source("global.R") somewhere in the beginning of app.R (for example just after the line library(shiny)).

We can now run the app and it should load without errors. When we click Run App (only visible when the file app.R is selected – this won’t be visible when the file global.R is on the screen in RStudio).

At this point, the app should load and look exactly the same as it did originally.

If Run App failed, then it is most probably due to the fact that R cannot find the file global.R – hence check if they both are in the same directory.

The functions of global.R are now loaded, but not yet executed. To get this done, we need to modify the file app.R. We will load the data and immediately prepare some key data, such as the last date and numbers:


gc_dts_countries <- get_data()
gc_countries     <- c('ALL', unique(gc_dts_countries$Country))
gc_last_date     <- max(gc_dts_countries$Date)
gc_first_date    <- min(gc_dts_countries$Date)
gc_impact_types  <- colnames(gc_dts_countries)[3:6]

Again, we can run the app and check if all works fine. It should load slower and produce more output in the Console (lower left workspace in RStudio), but still look as it did originally.

We will now add a country selection and present some basic data related to that country. To get this done we need a few thing simulteneously in place: the user interface and the server functionality.

We will now delete the functions of the skeleton app, add the functionality of selecting a country and show an elegant box with the latest number of infections in that country. When the app starts, the selection is ALL (the sum of all countries).

In the user interface, ui, we replace the sliderbard with a selectInput


selectInput('sel_country',
             "country:",
             gc_countries,
             selected = NULL,
             multiple = FALSE,
             selectize = TRUE,
             width = "100%",
             size = NULL)
      )

and in the body of the app we replace the plotOutput() function with a valueBoxOutput() as follows:


valueBoxOutput("infected_box",  width = 12)

In order to be able to output the value-box with the name “infected_box”, we need to create an output variable of the same name. So, we replace all the content of the server function with:


    output$infected_box  <- shiny::renderValueBox({
        last_nbrs <- get_last_numbers(d = gc_dts_countries, 
                                    theCountry = input$sel_country)
        x_frm <- last_nbrs$Confirmed,  big.mark = " ", scientific = FALSE)
        valueBox( x_frm, "Confirmed", icon = icon("ambulance"), color = "purple")
    })

The app will now look as follows:

Figure 1: Our app is still very simple, but it does display the latest total number of Covid-19 infections.

First we will change the basic layout of the app from standard bootstrap to the elegant dashboard interface of shinydashboard. This is done by loading the library shinydashboard and replacing the main function in the user interface from fluidPage() to dashboardPage(). We also add a dashboard header and replace the sidebard with the dashboardSidebar(), add a title, etc. The first lines of the ui in the file app.R look now as follows:


ui <-  dashboardPage(
    skin = 'red',
    dashboardHeader(title = "My Covid Dashboard"),
    dashboardSidebar(
      sidebarMenu(# nice layout + indication which one is active
        # Header:
        h4('Selected Country'),

        # selct the country input:
        selectInput('sel_country',
                        "country:",
                        gc_countries,
                        selected = NULL,
                        multiple = FALSE,
                        selectize = TRUE,
                        width = "100%",
                        size = NULL),

        # a separator:
        hr(),

        # the menu items:
        menuItem("Main Page", tabName = "dashboard", icon = icon("th")),
        menuItem("Forecast",  tabName = "forecast",  icon = icon("sun"))

        ) #close sidebarmenu()
        ), # close DashboardSidebar

The app will now look as follows:

Figure 2:Now, the app looks like a real dashboard, but we only started to work on it.

Now, we have a fully functioning app that loads the latest covid data and displays just one number. We will now expand the app so that it beocomes more meaningful. Along the way we will also improve the code slightly.

Wit the following code we prepare more pages, enrich the menu, and make it look nicer.


 dashboardSidebar(
      sidebarMenu(# nice layout + indication which one is active
        # Header:
        h4('Selected Country'),

        # selct the country input:
        selectInput('sel_country',
                        "country:",
                        gc_countries,
                        selected = NULL,
                        multiple = FALSE,
                        selectize = TRUE,
                        width = "100%",
                        size = NULL),

        # a separator:
        hr(),

        # the menu items:
        menuItem("Dashboard", tabName = "dashboard", icon = icon("th")),
        menuItem("Forecast",  tabName = "forecast",  icon = icon("sun"))

        ) #close sidebarmenu()
        ), # close DashboardSidebar

We can test the code by clicking Run App, however the new buttons won’t work yet (they do not generate an error message either).

We need to connect each tabName to the content in the dashboardBody(). This can be done by adding a tabItem with the corresponding name in the dashboard body (for example wrapped in the function fluidRow() – which produces a Bootstrap row of 12 unit lengths: this creates rows that turn into columns in narrower screens).

Here is a minimal example:


dashboardBody(
      fluidRow(style='padding:5px;',
        tabItems(
            tabItem("dashboard",
                    h1('The Dashboard'),
                    valueBoxOutput("infected_box", width = 6)
                    ),
            tabItem("forecast",
                    h1('The Forecast'),
                    HTML('TBA') # we will add this later
                    ) # close tabItem
        ) # close tabItems
      ) # close fluidRow
    ) # close dashboardBody()
) # close dashboardPage

Again we can test the app at this point, and now we can select other pages in the menu. For example, if we choose Forecast, then we will see the message “TBA”. So, the layout framework is operational and we can focus on the content.

Now, we need to develop the code (server function) and html5 ui for each tabItem. First we focus on the main dashboard. Now it is the following block.

We can now copy the following code in the beginning of the dashboardBody():


    dashboardBody(
      div(style='padding:5px;',
        tabItems(

            tabItem("dashboard",
                    h1('The Dashboard'),
                    column(6,
                      valueBoxOutput("infected_box",  width = 12),
                      valueBoxOutput("sick_box",      width = 12),
                      shinydashboard::box(
                        width = 12,
                        title = "Sick as a percentage of Confirmed",
                        gaugeOutput("sick_gauge",  width = "200px", height = "200px")
                      )
                    ),
                    column(6,
                      valueBoxOutput("recovered_box", width = 12),
                      valueBoxOutput("death_box",     width = 12),
                      shinydashboard::box(
                        width = 12,
                        title = "Deaths as a percentage of Confirmed",
                        gaugeOutput("death_gauge", width = "200px", height = "200px")
                      )
                    )
            ),

               tabItem("forecast",
                    h1('The Forecast'),
                    HTML('TBA') # we will add this later
                    ) # close tabItem

        ) # close tabItems
      ) # close fluidRow
    ) # close dashboardBody()
) # close dashboardPage

Note that:

h1() generates a level 1 header (it is the heading of the first tabItem)
We choose a layout in columns. Each function column, takes as first argument the width (e.g. column(6, ...)). Rember that the total width at each level is 12. Even if within the column of width 6 we start a new column, then this will again get a maximal width of 12.
Some items are wrapped in shinydashboard’s function box(). This draws a box around the items in it, but also allows to add a title to the items.
Each column has two value-boxes and one gauge plot. These still need to be generated. The code will work, but the plots are not displayed till we add in the server function the corresponding output variable.

To create the dynamic output that for which we created the output framework, we need the following code. Note that we decided to lift the data genearation for one country out of the renderValueBox() function, because we need it in every value box.

To make this data respond to the user’s input, we need to wrap it in the function reactive(). Later, we can refer to it as a function (last_numbers()). Since it is a named vector of values, we can get the last number of confirmed infected people for example as last_numbers()$Confirmed.


    # We extract only once the last numbers of the selected country:
    last_numbers <- reactive(
        get_last_numbers(d = gc_dts_countries, theCountry = input$sel_country)
        )

The app is no fully functional and the first page will look as follows:

Figure 3:The first page of the dashboard is finished.

All that rests to do now the second functionality: generate a forecast.

For the forecast, we will start from the server function. This will make clear what we want to do and what is needed in the ui.


    # The data for the selected country:
    d <- reactive(get_dts_country(gc_dts_countries, input$sel_country))

    # The forecast plot:
    dd <- reactive(add_forecast(d(),    # timeseries for the given country
                       input$sel_data,  # e.g Sick, Confirmed, etc.
                       input$dte_start, # starting date for the data to use
                       input$f_nbrDays  # number of days to forecast
                   ))

    # Prepare the plot for output:
    output$plot_frct <- renderPlotly(plot_forecast(dd(), input$sel_data))

This makes clear that in the input we will need the following input variables:

sel_country: which country is selected. We already have that one. This one is defined in the side-bar of the dashboard and is considered as an over-arching choice for the first tabItem (“dashboard”) and the second (“forecast”))
sel_data: the data to be used (confirmed, sick, deaths or recovered)
dte_start: the starting data. As the pandemic evolves, the dynamics evolve and we cannot expect that our simple forecast (based on an exponential model) will make sense on all data
f_nbrDays: the number of days to forecast

If we would like to compile the app, it will fail. Going through the error message, you might find that we miss the function plot_forecast(). This function can be added in the server function or in global.R We decide to add it in the global.R file, beasue it does not need reactive code and is rather lengthy.

Add the following to the global.R file:


add_forecast <- function(d,             # timeseries for the given country
                         theRow,        # e.g Sick, Confirmed, etc.
                         dte_start,     # starting date for the data to use
                         f_nbrDays = 14 # number of days to forecast
                         ) {

# prepare the data
dd <- d %>% filter(Date > dte_start)
dd <- dd[complete.cases(dd),]
x <- dd$Date
y <- dd[[theRow]]
n <- 1:nrow(dd)
N <- max(n)
dd <- cbind(dd, n, y)  # we add a column y (universal name for the dependent variable)

# find good starting values
theta.0 <- min(y) * 0.5
model.0 <- lm (log(y - theta.0) ~ n)
alpha.0 <- exp(coef(model.0)[1])
beta.0  <- coef(model.0)[2]
starts  <- list(alpha = alpha.0, beta = beta.0, theta = theta.0)

model <- NULL
try({
  model <- nls(y ~ alpha * exp(beta * n) + theta,
               data = dd, start = starts,
               #algorithm = 'port',
               nls.control(maxiter = 1000, minFactor = 0.000002, tol = 1e-03))
}) #try

if(f_nbrDays > 0) {
  df_frcst <- tibble(
    Date = as_date((max(dd$Date) + days(1)):(max(dd$Date) + days(f_nbrDays - 1))),
    y = NA,  #NOTE: we use y instead of Sick, Confirmed, etc.
    n = (N + 1):(N + f_nbrDays - 1)
  )
  dd <- bind_rows(dd, df_frcst)
}
#df <- df %>%  dplyr::mutate(Forecast = predict(model, newdata = df)) ## fails ... Error: Column `Forecast` must be length 1 (the group size), not 74
dd$Forecast <- predict(model, newdata = dd)
return(dd)
}


# ----- return the plotly plot
plot_forecast <- function(dd, theRow) {
  plot_ly(dd, x = ~Date, y = ~y, type = 'scatter',  name = "Daily people sick",
          mode = 'markers', marker = list(size=8)) %>%
    add_trace(x = ~Date, y = ~Forecast,
              mode = 'lines', type = 'scatter', name ='Model Estimate', line = list(width = 3), marker = list(size=1)) %>%
    layout(
      title = paste("Forecast for:", theRow),
      xaxis = list(title = "Date"),
      yaxis = list(title = "Number of People"),
      #autosize = TRUE,
      showlegend = TRUE)

As one can see from the code, the functionality works in two steps. First, we define a function that fits the model, then we define a function that uses this model to visualise it.

Again the app can be opened now, but the new tab will still not show up. To visualize the forecast, we need to build the ui.

We will now need to replace our placeholder for the forecast output in our ui. So, replace


            tabItem("forecast",
                    h1('The Forecast'),
                    HTML('TBA') # we will add this later
                    ) # close tabItem

with:


            tabItem("forecast",
                h1('The Forecast'),
                fluidRow(
                    shinydashboard::box( #input of extra variables
                        title = "Make your choice",
                        selectInput("sel_data",
                                    "Data:",
                                    gc_impact_types),
                        dateInput('dte_start', 'Use data from:',
                                  value = gc_last_date %m-% months(1),
                                  min = gc_first_date, max = gc_last_date,
                                  format = "yyyy-mm-dd", startview = "month",
                                  weekstart = 0, language = "en"                                                          ),
                        sliderInput("f_nbrDays", "Number of days to forecast:",
                                    value = 14, #the default is
                                    min = 1, max = 90,
                                    step = 1, round = TRUE)

                       )
                    ),
                fluidRow(
                    shinydashboard::box( #input of extra variables
                        width = 12,  # default is 6
                        title = "The forecast",
                        plotlyOutput('plot_frct',
                                     width = "100%", height = "350px", inline = FALSE
                                     )
                                    )
                    ),
                    ) # close tabItem

Finally, the app if finished and the forecast page will look as follows:

Figure 4:The forecast page.

With little effort we created a fully functional app that interacts with the user and uses the user’s input to calculate and visualise things that the programmer might not have expected (e.g. special choices of dates), or that look different every day (since we pull in new data at the start). Obviously there is still a lot that can be improved:

the mathematical model in the first place needs improvement
a wider range of models would make sense (e.g. a logistic curve, and ARIMA model, etc.)
and above all there is much, much more possible, but that dashboard is already available: see Philippe’s Covid19 dashboard

Anyhow, I hope that this gets you started on sharing your data, analysis, and insights.

How to become a data scientist

2021-01-04T21:49:00Z

Maybe you have read my latest book about data science or you have seen my courses about data science on my YouTube Channel, and it appeals to you to become a data scientist yourself. Then this post is for you: I will share some ideas about what makes a good data scientist and how you can improve your qualities.

Data Scientists are in demand, much of the new jobs and much of the new economy seems to be related to data science. It seems as if data science and its various sub-fields such as deep learning, big data, statistics, modelling, etc. are supporting a new wave of development and wealth creation.

Data science is a worthwhile pursuit, and is a road to a rich and interesting career with amazing growth opportunities. It is, however, not the most obvious thing to learn. That is because data science does not a narrow scientifc field. Data science is the art and science of making (business) decisions based on data, hence one need computer skills to access and manipulate the data, business insight to know what data and how to manipulate it as well as the insight what analytics make sense, mathematical knowledge to run the model, and communication skills to convince other people about the results.

Once can say that data science as the art and science to use data to create actionable insights. As such, data science is the interdisciplinary scientific domain that lies at the intersection between mathematics/statistics, computer science, business management, and psychology … all this within an ever changing landscape.¹

In order to be succesful in data science, you will need to have a basic understanding of all of the sub-domains, excel in a few and make this your personal trademark. These sub-domains break down as follows:

Mathematics/statistics: a good understanding of basic notions is maybe more important than a superficial knowledge of more advanced techniques.² – The best ways to build up this mathematical knowledge is via an academic training (STEM subjects are obvious candidates and are the ideal starting point – it is rather hard to acquire this knowledge later if you don’t have a solid base).
Computers science: understanding databases, software design, OO, and being able to write code in SQL, R and Python is essential and should be considered as minimal requirements.³ Programming is best learned on-the-job: simply start a project, find solutions to all your problems and don’t give up. You might also find my movies on YouTube useful.
Business Management: understanding the business model in your sector is essential to produce analysis that make sense and can be used by the management. A data scientist has a purpose and a mission, you need to understand that mission and always keep an eye on the bigger picture. This business insights is something that can be acquired by reading about the subject, studying applied economics, but years of experience will always add value.
Psychology, Game Theory and Decision Science: your job is not done by crunching numbers: you will need to present them, suggest solutions, etc. … and most of that is only possible via solid teamwork. So, you have to be able to work as a team, and making sure that you produce the most impactful analysis possible that is digestable and actionable. Reading up about multi criteria decision analysis or game theory might tick certain boxes, but beging effective is something that you will need to work on permanently. For example getting and giving feedback will help you grow.
An Open mind and willingness to learn: the domain of data science is an ever moving landscape, some skills that are explained above only help you to move towards the top, but being a data scientist means that you need to be willing to learn.

So, you will want to start by studying something that is of interest and can be used as an angle of approach to data science. Any STEM subject will do, as well as econometrics, applied economy and business analytics.

Then you need to read up about the fields that were less prominent in that study. I have tried to find a book that could help you on that journey, but could not find one … so I wrote it myself The Big R-Book: From Data Science to Learning Machines and Big Data. The book is about data, and how to use it successfully in a private company. The book is not only a technical work, it aims at increasing your personal brand and value, as well as shareholder value. You can learn about statistical models, muliti-criteria decision analysis, machine learning, artificial intelligence, big data, creating interactive websites, automated presentations, speed up code, program the GPU, etc. all while learning to code in R. The book takes a pragmatic stance and gets you started. It covers the whole dat cycle from data bases, importing data, wrangling data, to modelling, reporting and even elaborates on big data, interactive websites, etc.

Whatever route you choose to start, I’m sure that you are in for a fantastic journey in a fast growing field. I wish you loads of success and a rewarding career!

The Start of My Youtube Channel

2020-11-10T13:26:00Z

Dear Reader,

I started a Youtube channel, where I will post in the first place information about my newest book (eventually a series of trainings).

Visit the channel here.

Enjoy! Philippe

Working from Home on Steroids

2020-03-19T08:26:00Z

Today we face a novel virus, SARS-COV-2, and the disease that it causes, Covid-19, spreads fast around the globe. Most countries decide prioritize people’s lives over economic welfare and decide to enforce some form of social distancing to slow down the spreading of the virus. This means that people work more from home. It is still possible to forge strong teams that hold together, but it requires good intend, respect and an effort from both team-leader and team-member.

During the last decades, we gathered a lot of experience working while with virtual teams that span different countries and multiple time-zones. Here are a few hints that can make a big difference, and turn this period of hardship in a period of opportunity, self actualisation, productivity, and personal growth.

Clarity is the quintessence of good management. Working from home is even more predicated on clarity from both team-leader and team-member.

Be clear on what you can and cannot do.
Be clear on what you need in order to be successful.
Speak up if you’re stuck, if you feel bad, even personal things matter, and yes: your team leader does care.
Provide feedback to everyone, and do not forget that your manager will need now feedback more than ever.

The cornerstone of a good functioning team is trust, and we tend to trust people that we see in person and meet often. Working remotely, we cannot meet in person and hence need to make sure that team members have ample opportunity to see each other and communicate.

Fortunately, technology helps us to stay connected, and offers us the next best thing to a meeting in person: the video conference. To get the best out of it, you will want to:

Make sure that the camera is on. This might be the only face that your lonely colleague will see for a while.
Make time for small talk and don’t be afraid to bring something personal to the meeting. Do not be ashamed that your pet or kid wants to be on the video conference, realize that you are doing an effort to get work done while they are around. You will be surprised how your team-members will be understanding.
Organise virtual coffee breaks without specific agenda, but just allow people to talk. Why not spice up the virtual break with a fun contest (drawing, best dressed, a fitness challenge, etc.)
Start from the assumption that team-mates are people worthy to be your friend. Start an online, informal discussion group and see what happens. Have fun, but keep it respectful.

It is always important to deliver what you commit to do, but while working from home makes it becomes even more more important. Think about it this way: even in these days, there are managers that believe that productivity will go down when you work from home: prove them wrong! Prove them wrong and earn the right to work more from home.

Make sure that you understand what needs to be done. Re-read the section on clarity and reflect on it regularly.
Make sure that the task allotted to you is realistic (be clear and frank if you cannot do it).
Even when the two previous point are satisfied, things can go wrong. Make sure to reach out for help or escalate early.
Never, ever get stuck. Be your own coach and reflect often, that will help you to recognise early when your task does not go as planned.
Be a problem solver, bring solutions to the meeting, not problems.

Accountability alone will not keep the productivity on a sufficient level, one must practice the fine art of being efficient and effective.

Avoid procrastination.
Plan ahead, set objectives and measure outcomes, hen celebrate successes.
Reflect often if what you are doing will work, is there a more efficient way, is it effective, etc.
Use team meetings and personal meetings with your manager to reflect on those issues. You will be surprised how effective it is to formulate the problem out loud.
It is O.K. to meet more when working with a virtual team, but try to keep meetings short.
Be efficient in meetings (do not change the subject before the decision or action is clear, summarize actions and decisions at the end of the meeting, stick to the subject and avoid elaborating when the decision is already made, etc.)

Working extended periods from home might blur the boundaries between work and private live. This is bad news for your private life, work and overall well-being. In order to be able to relax, body and mind need to see and feel a difference between work and relax.

Plan the week: What are the key things that you want to achieve this week?
Plan the day: Given the weekly objective, what can you do today to achieve that weekly objective? … and do not forget the coffee break
Dress for the occasion: It is ok, to opt for a dress code that is less formal than in the office, but changing clothes when transiting between working and private life helps the mind to change to the other mode.
Master the space: in the office you have a caring teams that make sure that workstations are ergonomic and functional. It is important to make sure that you can have an ergonomic setup of your work-station.
Mark your space: try to have your space to work. Even if your space is at the kitchen table, then it should be your workspace during the hours that you work. Ask family members to respect this, where possible, or integrate them in your work-experience. If that fails, discuss with your manager to adapt your workload.

An ergonomic workstation that allows you to switch between a sitting and standing position is not enough to stay healthy in long term.

Take regular breaks.
Stay hydrated, have a bottle of water next to you.
Make sure that your workstation is properly installed so that no joints would be strained much (straight wrists, straight angles in knees, screen high enough so that you do not have to bend your head, etc.). You know the drill of a healthy position after the computer. Now change it. Even the best position after a computer is not a position to use 8 hours straight. Move around, take some meetings on the phone and lay on your back or stand (decompress the spine), sit on a giant ball (not more than 15 minutes if you’re not used to it), walk with the phone, build a simple standing desk, … do whatever works but change position often. Especially take care of the following:
- I repeat: change position often.
- Do not work the whole day in front of a laptop that is not raised; invest in a detached monitor, keyboard and mouse.
- Do simple and light (stretching) exercises in a 5 minutes break. Start a meeting saying that you will stop 5 minutes early and be efficient.
- Decompress the spine after sitting an hour or so (stand, stretch, lay down, etc.).
Get into a workout routine: daily is best (every second day is nearly as good and even better when you’re over 50). Whatever you do, do not overdo it and seek advice.
Indeed, be careful: most exercises can be dangerous if not performed correctly or with too much weights. Ask the team, there must be someone that is happy to share information.
Set your goals. What do you want to obtain during this quarantine? Improve stamina, loose weight, or build muscle? Define your goal, set a plan, and commit to it … just as you would for work.
Why not combine it with informal team-meetings? Stream workouts and do it together, set a challenge, etc.

All the things mentioned under the heading “get organised”, will help you to keep a healthy work-life balance. However, in these trying times we might want to pay extra attention to other people too and we should seek to fight burnout and social isolation long before it occurs. Also the things in the last section might help. Additionally, you might consider:

Meditation, mindfulness, coaching and mentoring. Reach out to find a partner for whatever you choose.
Start a project to keep your mind busy (find a pen-pal, learn a new programming language, read a good book, pick up welding or knitting, etc.)
Whatever project you choose, talk about it, share your project with team members, find like-minded people and socialize. For example start a language-partner program in your company, a robotics club, … even a cycling club will work (just keep a healthy distance or stream your cycling on video).
Work out (see also the section on physical health) – working out will help you to feel better, sleep better, and be a better version of yourself in general.

Generally, extroverts have an edge when it comes to making career, and functioning in an open floor system. Today, the extroverts will face a more challenging time. While the introvert welcomes the extra time that is not spend on commuting and will naturally fill it with introspection and interesting projects, the extrovert might lack energy due to thinner or more shallow social interactions.

Social distancing, closing of pubs, movie theatres, and schools is bound to create problems for many people. Some people will have to cope with spending a lot of time with a small group of people and others will face the looming abyss of loneliness.

Be perceptive for other’s people’s needs. Do not assume: ask. Even when you’re not in doubt: ask, do not assume.
Parents with small kids in a small apartment will have a hard time to focus on work, be mindful and check in with them, offer to take over some work, etc.
People that are lonely need friends now more than ever. Be that friend, reach out to team-members and let them talk, share something of your own, be not afraid to share your emotions. However, do “stay on the right side of creepy”, and respect each other’s privacy.

You made it this far. Great! That’s a first and important step. Now, the best way to go is to reflect on this article, make your personal top three of things that you did not do and would like to do. Then write them down as your personal goals. Keep the goals next to your working station and try to do it. Then regularly revisit this article, and update your top three: your route to continuous personal improvement is on a good track … time to set more ambitious goals.

Good luck!

Philippe

How to create a static website with Mynt and Bootstrap?

2015-10-24T20:08:00Z

You are maintaining a website for your business or club? You need a little more than a simple html page but you don’t need all the functionalities of a CMS such as Drupal or Joomla? Or you’re fed up with the vulnerabilities of most CMSs? If your website does not require exensive user interaction for contribution to then a static one can be a simple and effective alternative. My personal website de-brouwer.com is generated with the “Mynt” and to allow a responsive design I used Twitter’s Bootstrap framework.

This tutorial will help you getting started.

Of course we assume that you’re using Linux, and for this tutorial we will use a Debian repositories to install the necessary software.

The nice thing of working with Mynt is that you don’t have to install a webserver such as Apache on your box in order to test the website, mynt will be able to show the webpage via the command mynt serve DIR, but more about that later.

first you will have to install mynt (which will will need pip)
```
sudo apt-get install python-pip
pip install mynt
mynt --version
```
The last line should show output similar to mynt v0.3.1. If you would not see this, then something went wrong and you will have to refer to the relevant websites mentioned above.
Second you will need a copy of Boostrap. Simply go to the download page and select the package that suits best your needs.

Open a terminal and go to the location where you want to create the website. We assume that the website is called my-site.com and that the source code will reside in the sub-directory src and the result (html) files will go in a directory docroot.

The init command will generate the structure that you need to generate the website (with some simple content), the gen command will generate that website.

In order to see the result (ope a second terminal window by typing ++. In that second window we start the built-in webserver of Mynt.

Then open any browser and go to 127.0.0.1:8080 in order to see the result. It should look like this:

This terminal window (and browser window) can be left active and next time we simply update the code, run gen and refresh the page in the browser to see the update

Now, we can go back to the first terminal window and first explore what files were created there by the init command:

These files are the following

archives: this directory hols an template for the blog archives.
_assets: this directory will hold all impages, and CSS files … and hence also our bootstrap code. In this directory you can use any structure that you like (but note that this structure will be replicted in the docroot directory).
config.yml: this file is in yaml-format and holds the configuration of the website.
feed.xml: this xml template will generates an Atom feed for your site’s blogs.
index.html: this is the home-page for your website
_posts: here you can put all the blogs that you want to publish in Markdown format.
_templates: this directory holds all the layout templates are. All html pages will typically declare “extend layout.html”, this will ldoad the “layout.html” that sits in this directory. There are also templates for individual blog-posts (“post.html”) and an archive page (“archive.html”).

As you will have notice Mynt assumes that you create a website that is geared towards a blog-website. While this might seem a limitation, it is actually very flexible. The other –static– parts can be generated in a simple way, and the only part that needs to be updated frequently (with automatic generation of overview pages, pages per topic, tag-clouds, etc.) is the blog-section. Of coure the logic for blogs can be re-used to function with “news” for your website.

You will also have seen that the content of the html files is in a specific format: those files are Jinja templates. Refer to the Jinja Template Designer Documentation to learn more.

Step one is of course to download Bootstrap.

Go to the http://getbootstrap.com/getting-started/#download
You will get a zip-file that should be extracted in the _assets directory (so that you have _assets/bootstrap/css, _assets/bootstrap/fonts and _assets/bootstrap/js)

The next step is making sure that your website uses Bootstrap. Mynt provides a flexible framework that is easy to modify. Simply edit the file _templates/page.header.html and replace the line refering to the css file with the two following lines:

Now, your website should look like this:

You will notice that this is the same content as in previous section but that the layout is dramatically different. Already now you have a website that is using Bootstrap. Now you can start experimenting and changing the content of the pages, creating new blogs, etc. However if you like a website that is less oriented towards a blog, then it is probably worth reading on.

If you indeed left the second console (with the mynt serve docroot open, it is now sufficient to recompile the website (run the command mynt -f gen src docroot in the first console) and go to the browser that has the page open and refresh the display (press F5) in order to see the results of your latest modifications.

At this point we need to make a few design choices. If you want to make a blogging site, then you can start adding blogs to your site and all is ready to go. However if you make a more general site, there is still some work. For the rest of the tutorial we will assume that we make a more general website where we will keep the blogs to be a flexible section for “news”.

The root index.html will be a showcase for the blogs. This is probably not what you need, so we move this file to a subdirectory and create another intex.html file.

Create now a new index.html file and add the following content

The menu of the website that we will build now will be simple: only contain the following items:

Home
News
Value Offer
Proof of Concept
More
Contact
About Us

For illustration purposes, we will have all of those items referring to the main page (index.html) except News, Proof of Concept and Press. Here the power of a tool such as Mynt becomes clear: in stead of copying the menu on all pages, we simply create it once and include it in other pages: when we then compile the website, Mynt will add it to all pages. To make it even easier to maintain, we will create a variable to store the menu and its URLs. Doing so, it will be easier to make changes.

First, create a new file in _templates, that you will call nav.html and give it the following content:

Now replace the content of the file _templates/layout.html with the following content:

Then add a new file _templates/footer.html with the following content:

This footer file can contain whatever you like (you can leave it blank if you wish). Note that this file will take the author form the src/config.yml file. So if you want to use it in this shape, you will need to update the config.yml file at this point.

Of course you will have noticed that this footer contains buttons for some popular social media to share your web-page.

Now create a few files:

Of course, the precise content of these files will have to suit your own needs, but here is some inspiration.

For the index.html file:

Now, your website should look more or less like this:

Try to resize the browser window and you will see that the layout changes as the screen gets wider or narrower. For example a narrow screen will make the menu collapse in to the “hamburger” icon.

The index files in more,about and poc should have a similar setup as the index.html file in the root directory.

The minimal setup would be (for the src/about/index.html-file):

Secondary files in those sub-directories should look almost the same (although you might want to adapt the breadcrumbs). For example the src/about/philippe.html could look as follows:

This is of course just the start, all the content and many other whistles and bells might be desired/needed. You will want to add something like Google Analytics (to gain insight in how many people are reading your pages), a search functionality that is limited to your websites, a favicon.ico file, etc. Naturally, you will want to use the power of Bootstrap in each and every page, but since that is the reason why you started reading this tutorial in the first place, I assume that you know where to find information about this.

I hope that this is a good start!

Good luck!

I hope that this tutorial is helpful. Please let me know if you find inaccuracies or if you think that it can be improved.

Trends in Asset Management in Europe via Five Questions

2014-05-18T22:02:00Z

Before answering that question it is probably wise to draw the attention to two little known facts that will help to understand what we’re actually talking about:

The European asset management market is dominated by institutional investors: roughly 75% of all assets are controlled by institutional investors. Insurers and pension funds and together control 42% and 33% is owned by other institutions. In order to win an institutional mandate one does not necessarily have to open another branch in another country, in any case institutional assets are stable and follow a strict governance.
The European asset management market is mainly a market of UK (36%), France (20%) and Germany (10%). All the rest is much smaller: the three followers up are: The Netherlands, Italy and Belgium with resp. 4, 4 and 2% of the assets. There are of course historic reasons that go back to the industrial revolution that started in the UK and conquered Europe via France, The Netherlands and Belgium. These early adapters had an important early starter advantage in cumulating wealth and are roughly 200 years later still among those nations that accumulated the most wealth.

These two facts combined means that the large players cover easily 90% of the market with 5 to ten locations (which they already have most probably). The new investments are only marginally contributing to the bottom line and are mainly investing in the future growth of certain regions. What one sees is that there is more and more competition of the large USA houses (the Blackrocks and State Streets of this world).

As argued before, distribution is key and the highest entry barrier. The cost of creating an asset manager is a containable cost, distribution might not at all be possible. The following approaches are used:

Creating an asset manger and its processes as well as hiring some key people is not so difficult: it’s a process that is containable and will typically cost you 10 country-average year salaries. Expect to need a team of 3 to 4 very experienced people to set up the thing and get the licence in Europe. Then you need of course to step up. These costs are containable and if distribution is working out then it is easily and fast earned back. The key in this approach is to set up a viable distribution. Therefore this approach is typically used if one owns distribution (insurance network, bank, etc.)
Buy an existing company. This is an attractive approach as it will ensure that you [a] ave a licence, [b] have some good people with local knowledge on board and [c] that there is some distribution in place. Also you can expect to have some assets and in many cases they appear to be rather sticky. In any case you will buy only if you believe that you have something to offer to the investors.
More and more since UCITS IV it becomes realistic to distribute products without having an asset manager in place. In most cases the local distributor will expect a local contact person or organization, but this can be your (small) sales wrap. This is more and more an attractive approach if you don’t want to become the unique choice of the mass market. Indeed in most countries becoming the choice of reference will require to have products that offer a local bias.

The answer to that question seems surprisingly stable. UK managers for the last decades tend to indicate that they pefer more exposure to hedge funds. Of course the reason is the search for alpha and diversification, both rational arguments to look at hedge funds.

However there are loads of “irrational arguments” that make people afraid. Many have tried to blame hedge funds for the crisis but all those attempts have failed. For some it would have been convenient if an industry that is weakly or not organized could be blamed so the attention on them would fade, and for others it was their insatiable hunger for controlling all into the greatest detail that drove them to try to blame hedge funds. Despite the fact that no evidence was found that hedge funds could be to blame, they got more than a fair share of regulation. This of course fuelled the anti-hedge fund climate and slowed down plans of institutional investors.

Another equally true explanation is that it is not so easy to invest in hedge funds. It takes a decade of errors and dedication to build up the capacity to invest in hedge funds. That’s the price that the Instutional Investors of the UK ar paying because of their choice to avoid funds of hedge funds and invest directly. One can therefore expect that investing more into hedge funds will remain high on the agenda of institutional investors for the next years to come.

If an asset manger wants to be successful, then there are three crucial things: distribution, distribution and distribution. Asset management products are sold based on confidence. Confidence is not something that can be gained via computers and electronic media. People need to “look someone in the eyes” to know if he/she is reliable. Further confidence is not build up fast: it takes time. This means that the local branch of a bank that served your father and that financed your baptising party has an enormous advantage over other channels. People tend to trust advisers when deciding about investment products. This is not only true for the the small investor who gathers 20K over his/her lifetime, this also holds for private banking. The private banking client might be much more knowledgeable about investments and money, still trust will be the basis and cornerstone of the relation.

So, if you don’t have your own distribution channel you’re simply one of the sharks in the “Red Sea”: you have a product that has no real differentiators, you cannot guarantee anything (not even that you can replicate past performance), … and to make things even worse statistically you’re bound to destroy value for your customer.

If you want to break through without a distribution channel then you must have an extraordinary product. However typically these extraordinary products are simply one bet on one trend. For example the fastest growing asset manger in Poland –Arka– in the period 2000 till 2007 had one strong focus: small cap companies. Every year they had the best performance of the whole market and his sales was the best … till the crash of the small cap companies when they showed a more than fifty percent loss to many customers. For half a decade they have set the trend and were the top-selling asset manager, all based on one lucky bet.

In summary, asset management is a business with seemingly low entry barriers, low capital requirements and can be set up with reasonable investment costs and lead time. However, it is very hard to create strong and sustainable distribution.

Part of this question could refer to companies and the other part to products. First the companies: it has simply been a dull place in Europe without major changes.

However, the asset management market is a very lively place and there are always companies doing good. For example we have a few strong hedge funds doing good business and for example Jabre Capital (Swiss based) is world leader in contingent Convertible Bonds (CoCo Bonds).

Coco bonds are a relatively new and exciting asset class. They are supposed to be part of a bank’s contingency plan. The idea is that these bonds would be a buffer between equity and capital. The idea is that if a bank gets liquidity problems that then those bonds are automatically (at least not at the decision of the investor) converted to equity or even considered as lost. That could be enough so that that bank should not rely on the state for a bailout. The legislation is still in its infancy as is the product itself. That means that there is a wide variety of different terms and conditions.

My personal guess would be that this asset class would grow fast and steadily. The logic behind is solid and the benefits are huge: a state would be able to ensure basic banking service without having to risk taxpayers money by investing into banks or granting them loans while no-one else does provide liquidity any more. The deeper –and more important– potential of this approach is that in the long run one can hope that this will spark a wave of de-regulation. That in its turn should

reduce the operational costs for banks and hence spark their role in re-allocating liquidity;
re-educate banks in the sense that they take their responsibility serious and not just depend on states to help them if anything goes wrong (one of those things to do is to use coherent risk measures and stop using the flawed Value-at-Risk that accompanied banks to their graves only a few years ago;
re-educate investors in thinking where to allocate money (ie. demand a higher risk premium from banks with worse numbers). This is essential in creating a healthy free-market climate. This disease –in its non-cured form– is the same bacteria that pushed countries like Greece, Spain, Portugal and Ireland towards taking more risk on board without significant interest rate hikes to compensate those risks (and warn the governments of the risks associated to their policies).

Of course in these times where the call for more government intervention is out of control and where law-makers and controlling authorities compete for journalists and public favour by adding truck-loads of detailed laws and rules it will take a while before the bank system will be allowed to cure after its grave illness. First we will have to see trough a period of ill advised over-regulation that both slows down economy and make the banks even more sick.

Tutorial to create a Fully Automated Open-Source Work-flow for Statistics and Data Mining

2014-03-30T20:32:00Z

It is said that the typical person of our times consumes in one day as much information as a typical person in the Middle Ages would digest in his/her whole life. And this is only the visible top of the iceberg: the information that we see and get. But beneath the surface of these calm waters lays an ever growing ocean of information: the information gathered by electronic machines. From the app in our mobile phone counting your steps and pulling your position from the GPS to the transactional system your bank collecting your payment behaviour: every second loads of data are generated. Each person generates daily multiple mega- or even gigabytes of new and unique information. In order to make sense out of this we need to understand the underlying trends and concepts that are concealed in vast amounts of date: in other words we “need to run some statistics”, or do some “data crunching”.

There are very good commercial systems available that will do exactly this. Probably the best known is the information age pioneer and industry standard: Statistical Analysis System (better known as SAS). It will make your wallet a few million Dollars lighter and you get an interface that is an interesting mix of programming languages that in the 1980s were already outdated (as in “not even Object Oriented”) and the most modern, intelligent interfaces. It will come with loads of support and it will allow you to have a dynamical view, or a pre-generated set of reports at your fingertips in your favourite browser. SAS can run on a portable computer, a server or a super-computer; it is powerful and reliable. And above all it is the industry’s standard: so one can safely think: “no one ever gets fired for buying SAS”.

Of course there are alternatives, like Sephen Wolfram’s “Mathematica”. This is a more modern tool that actually does statistics and reporting as something additional on top of its real calling: mathematical (symbolic) analysis. It will cost ten to hundred times less than SAS (typically under the $100’000), but you might want to combine it with a database system. Oh, and if you want to get a flavour: it comes free with the Raspberry Pi! So it will cost you less than your next lunch to get started.

And of course there is an Open Source alternative (or as is so typical for open source: there is a multitude of possible combinations and configurations that can suit your needs). I will not give an overview of possibilities or combinations (although that cannot be avoided to mention a few), but in this tutorial I will try to give you one working solution.

The idea is more or less as follows:

Collect data in a database (we’ll use MySQL as example, but eg. PostgreSQL is also great (and it is ACID compliant! – The relevance of this is that you will never need a function such as the sql.reset.query.cache() as mentioned in our R-code)))
Build a Data Mart data with SQL
Build our model (or perform statistical analysis) with R
Present data in PDF via LaTeX (or HTML)

This stack has a few nice aspects and advantages:

It can be fully automated (every day your company’s website can reflect the latest result without human interference!)
It is completely free (gratis) and open source (so you can be sure that there are no backdoors as you can download code, check and compile yourself)
From the start to the end we use tools that are quire ubiquitous and for each of them there is an active community to support you … also for free. That is of course only relevant if your Google cannot help you.

Step-by-step guide to install the open stack.

The first thing that you will need is a working computer with a good operating system. Everything below should work on a Windows, OS X or FreeBSD machine, but our guidance is for Linux. We assume that you have a working distribution on your PC, that it is booted up and that you know how to install packages (we assume a Debian where necessary to make the example concrete)

Next we need to install the specific softwares that we will use. If you installed for example Ubuntu Server Edition, then you will not have to install anything, however, if you started from the bare necessities, you might need the following.

# to be executed as root or insert "sudo" before each line
#-- MySQL, the database
apt-get install mysql-server -y 
apt-get install mysql-client -y 
apt-get install phpmyadmin -y 
apt-get install mysql-workbench -y  # if you prefer a GUI for manipulation for writing queries
# -- R:
apt-get install r-base -y
apt-get install r-base-dev -y
apt-get install r-cran-xtable -y
apt-get install r-cran-msm -y
apt-get install libmysqlclient-dev -y # this is needed for RMySQL !!
# -- LaTeX:
apt-get install texlive-latex-base -y 
apt-get install latex2rtf -y 
apt-get install latex2html -y 
apt-get install biblatex -y 
apt-get install texlive-lang-polish -y
# -- eventually the text-editor of your choice
apt-get install kdevelop, kate -y
# -- and also we want to use pdfcrop, and something to visualize our PDF output so we need also 
apt-get install pdftk   # needed for pdfcrop (and other goodies) but does not include pdfcrop itself
apt-get install okular  # the PDF viewer

Now you still have to download the Perl script “pdfcrop”“ from sourceforge, make it executable and copy it for example in /usr/local/bin:

cp pdfcrop /usr/local/bin/
chmod 755 /usr/local/bin/pdfcrop

By now you should have a working environment with all necessary software installed. The next step is to set up a database, get your data and import it in the database. One caveat is that MySQL needs to be able to find your file to upload and it should be allowed to read your files. For me the following worked:
```
echo "Please enter your sudo password"
sudo cp MY_DATA.csv /tmp
sudo chown mysql:mysql /tmp/*csv
```
Note 1: Sure, for smaller datasets you can use phpmyadmin to upload the data. For larger data-sets only the MySQL terminal will work reliably … and of course only the terminal can be be fully automated (see below how). Note 2: If you are working on a machine that has only a command line interface, then you might want to use vi as a text editor and of course you won’t visualize the PDF files … but apart from that everything below should work on terminal (you can do without a GUI!). Actually during development a GUI is most useful, but once you want to automate the flow nothing can compete with the CLI. You will have the comfort that the work-flow described here can be copied to a headless server for example that has no GUI.

Now, upload the data in the database. Of course the code snippet below is just a canvas. You will have to replace not only MY_DATABASE, tbl_TEST, etc. with your names, but also fieldnames, filenames, termination fields, etc.

USE MY_DATABASE;
DROP TABLE IF EXISTS tbl_TEST;
CREATE TABLE tbl_TEST
(
account_number CHAR(12) NOT NULL, PRIMARY KEY (account_number), total_balance_01 DECIMAL(12,2)
) ENGINE INNODB
COLLATE 'utf8_general_ci';
LOAD DATA INFILE '/tmp/data.csv' INTO TABLE tbl_TEST
FIELDS TERMINATED BY ';'
ENCLOSED BY '"'
LINES TERMINATED BY '\n'
IGNORE 1 LINES;

Typically you will want to prepare the data and create some fields that contain aggregated information, or otherwise logically deduced information (for example if you try to model a lending portfolio, you will want to create fields indicating if an account –at a certain month– was ever in a delinquent state). This is handy as it will reduce compute time further down the road. Another handy hint is that you can simply write all your SQL code in a text file (call it data/build_balance.sql for example) and you can from the MySQL console invoke it by typing
```
source data/build_balance.sql
```

And we’re ready to link our database to R, or otherwise stated access our database from within R. To achieve this, do the following.

Get the necessary packages and load them:

install.packages('RMySQL')
library(RMySQL)

Then, I would advice to create a text file and store the following functions:

# -------------------------------------------
# -- function to open the data base
sql.login <- function(theUser = "root", thePwd = "", theDB = "MY_DATABASE", theHost = "localhost")
{
if (thePwd == "") {
cat("Enter MySQL root Password: ")
thePwd <- readline()}
con <- dbConnect(MySQL(), user=theUser, password=thePwd, dbname=theDB, host=theHost)
con
}
# -------------------------------------------
# -- close the database-connection
sql.close <- function(con) {dbDisconnect(con)}
# -- get the sql data 
sql.get <-function(ssql,NBR=500000)
{
rs <- dbSendQuery(con, ssql)
data <- fetch(rs, n=NBR)
huh <- dbHasCompleted(rs)
dbClearResult(rs)
data
}
# -------------------------------------------
# -- execute a query that does not return anything (used for UPDATE, CREATE, etc.)
sql.run <-function(sSQL)
{
rs <- dbSendQuery(con,sSQL)
}
# -------------------------------------------
# -- reset query cache 
sql.reset.query.cache <- function ()
{
system("sync && echo 3 | sudo tee /proc/sys/vm/drop_caches")
rc <- dbSendQuery(con, "RESET QUERY CACHE;")
system("sync && echo 3 | sudo tee /proc/sys/vm/drop_caches")
}

If you call this file for example functions_sql.R, then you can load this functionality in R with the command 2. Formatted tables. For example use the print function for the xtable object:
```
source('functions_sql.R')
```
That allows you to re-use code in more than one project. So, what you probably will want to do is build a file that loads all your packages, loads functions_sql.R and then runs all your analysis. We’ll refer to this file as calc_Balance.R.

Now the database can be used from within R.

con <- sql.login(thePwd = "XXXXXX",theDB="MY_DATABASE")
bal <- sql.get("SELECT total_balance_01 from tbl_TEST WHERE 1;")
summary(bal)
plot(quantile(bal))

This allows us to use the strengths of R combined with the strengths of our data mart to do our analysis, build our model or whatever we’re interested in or paid for. The trick is now to make sure that the output is generated in a format that can be picked up by LaTeX. There are at least three simple possibilities.
1. Pre-formatted text. This might not be very elegant, but it is fast and it "just works”. This consists of transferring how something looks in the R-console to our report. For example
```
sink("./tex/inc/summary.tex")
summary(bal)
sink()
```
2. Formatted tables. For example use the print function for the xtable object:
```
install.packages('xtable')
library(xtable)
M <- matrix(c(1,2,3, 11,12,13), nrow = 2, ncol=3, byrow=TRUE, dimnames = list(c("row1", "row2"),c("col1", "col2", "col3")))
sink("tex/inc/matrix.tex")
x<-xtable(M,align="|lrrr|",digits=2, caption = "my caption.") 
print(x)
sink()
```
3. Graphs. Also here it is sufficient to print the plot to a file that can be loaded by LaTeX. For example:
```
install.packages('ggplot2')  # this has to be done only once!
library(ggplot2) # this should move to the headers of calc_Balance (or better in a headers.R that is called from there)
d <- data.frame(bal)
p <- ggplot(d, aes(x=bal)) + geom_histogram()
ggsave("tex/img/graph.pdf")
```
Now these elements can be added to you LaTeX (or HTML) report. Note that we assume that the calculations are done in a certain directory and that in that directory we have a subdirectory tex in which we have img and inc. A minimal skeleton could look as follows:

And name this file for example model_doc_Balance.tex

Typically one will want to re-run a model and its validation information (statistics such as Smirnov-Kolmogorov test, etc.) on a regular base when new data comes available.

Now that we have the work-flow in place it can be automated end-to-end and even starting the workflow can be automated. To complete this it is sufficient to create one bash file that will do all previous steps sequentially. Assuming that we would for each of the previous steps the following files,

data/prepare_Balance.sh (to do the initial cleanup of the data via sed and awk for example – eg. insert comma’s in a fixed width file, etc.)
data/import_Balance.sql to prepare the data fields that we want to use.
calc_Balance.R to do all the statistical analysis (eventually using a header file to load packages and our functions_sql.R file, a configuration file to define the specific parameters and names) and spitting out the text and graphs in the tex/img and tex/inc subdirectories.)
tex/model_doc_Balance is your model documentation booklet in LaTeX
tex/pres_Balance is the presentation about the model (slides in the “Beamer” class of LaTeX for example)

and assuming that you use Okular to visualize the PDF files in the last lines, then this “do-everything-file” could look as follows:

Note: The first part in this file prepares the compilation of the LaTeX markup files. This is a little more complex than simply “latex myfile”, because it is prepared to allow you to use the package nomenclature, build an index, build a table of contents and have a reference system in place. The idea about the multiple runs is that we want to be sure that one call to our code gets all page references right (maybe I make a separate tutorial about this one).

Assuming that the shell file from previous point is called “calc.sh”, we could run then our project with the following command:

Linux is not only market leader on the small computers (routers, storage arrays, phones (via Android) and leader among the biggest computers on this planet, it is also a versatile environment that allows you to control your system the way you want. It is for example very simple to fully automate the initiation of all calculations.

Typically it is sufficient to add it to your relevant crontab file (eg. /etc/cron.monthly) or refer to the documentation of your distro to fine tune cron (for example askbutu for the Ubuntu distribution.

Free and Open Source Software (FOSS) is for people who value freedom, and this workflow can be moulded to suit your specific needs to any extend. For example

You can include the downloading of new data when it should be available.
You can email a report by using the mail command (on Ubuntu that might require to install first mailutils: sudo apt-get install mailutils)
You can replace LaTeX markup with Markdown and automatically create a website using Jinja2 and Mynt for example (those are also the systems that helped creating this website). In that case you can even add to your automation file some checks if it worked fine and then us rsync to upload the new html files to your webserver.
Of course you can replace MySQL with PostgreSQL, R with Octave, kate by vi, etc.

I hope that this tutorial is helpful. Please let me know if you find inaccuracies or if you think that it can be improved.

Workshop on Teamwork and Leadership

2013-09-15T18:27:00Z

Recently I did for my employer a workshop on Teamwork and Leadership. I loved every second of it and so did the participants. The feedback was very encouraging.

Druring my carreer I’m in a people managers role for about 20 years now with many C-level function in different countries, on different levels and in different companies. I made a lot of mistakes! But I also did some things that really worked out and recently my boss gave me the works of Patrick Lencioni and a whole new world opened for me.

I started to realize why certain things worked out well and why certain things did not work. It became a passion … that I love to share of course and recently it culminated in this workshop. The workshop does two things:

building understanding and trust with the MBTI personality indicators as a guide and
building on that trust to lay the foundations of a performing team.

Such workshops become very personal (and sometimes very emotional too), and to make it fit my own personality I decided to package it as “executive training”. With that I mean that it was highly personlizized and it is designed to be done with the group that should function as a team. The training is actually moulded in function of the personalities of the people present, also the materials that afterwards everyone gets are highly personlizized to support that particular team.

Download Flyer

Maslowian Portfolio Theory

2012-02-18T11:49:00Z

Finally, I finished my PhD. While it was an enjoyable process and I really learned a lot on the way, it felt like ages. I wanted so much to finish the Ph.D. before my second child was born. Now he is almost a year old. A big thank you to my wife, Joanna, for all the support and my gratitude to my promotor, Freddy Van den Spiegel.

Of course, making a Ph.D. is a rewarding experience and it really teached me a lot about scientific work, the scholastic method, coherent investments, etc. … but above all: I’m on a mission.

I’m on a mission to improve advise about investments. I truly and honestly believe that almost all banks, robo-advisers and agents do not a good job.

FINRA in the USA and MiFID in Europe require investment advisers to make clear if they give advice or not and if they give advice they must take into account the educational background and investment goals of the investor. This makes a lot of sense, but at best they do an awkward effort to fit liquidity into an old concept that is void of investment goals.

The main thinking about investment advice goes back to utility functions, and Markowitz' mean variance portfolio theory. This approach has the following issues:

It requires to consider all investments in one portfolio and optimize this. This implies that the house, the shares, the savings account and the cash to buy a sandwich need to be optimized in one bucket. This makes –for humans– no sense. Humans have needs that need to be addressed separately, they need clarity on what is achievable for what goal.
There is only one investment horizon in that optimization. What horizon makes most sense when we put a house that we plan to keep forever together in one bucket with cash we need tomorrow? There is no answer. Typically bankers assume that nor the cash of the house is part of the equation (is not considered as part of the investment portfolio), the apply an arbitrary horizon of 5 years … probably because they see five fingers on one hand.
There is no way to connect this result to reality and investment products in an objective way. In fact, we need the –not so intuitive– concept of “volatility” to make the link. This results in arbitrary matching of a an arbitrary safe portfolio to an arbitrary level of volatility, an arbitrary matching of an arbitrary level of high volatility to a dynamic portfolio and some kind of arbitrary scaling in between.
Most bankers optimize with a forced percentage of risky assets (shares), instead of cash a would be required by the CAPM theory. So, the optimization is arbitrary and subjective.

In summary this is a disaster for a generation. Markowitz' theory works fine when the investor is just gambling with money that he or she does not need (read: “is so rich that he or she never need to worry about subsistence”). This made sense in 1952, when Markowitz wrote his insights down and it was indeed as step in the right dthat irection.

However, today the world is a different place and we are all part of a generation that unlike any before us:

Will see that states will be unable to provide subsistence out of their balance sheets, because of increasing medical costs, increased longevity and decreased natality.
Hence, you will have to invest for your own retirement — this is no longer a choice.
Therefore, millions of people that will invest will not be so rich that they never have to worry about subsistence and they will have to worry about what to spend for what goal. They will have concrete questions like “How much can I spend on this dream holiday and still send my daughter to a good university in 5 years?”

The traditional theories have no answer and the states will have no cash. That’s why I think the sacrifices were worth it. And I apologise to my family, for it is not over, it is merely a start.

Well, we have on the one hand Markowitz' theory that tells to put all assets in one bucket and on the other hand the reality that no-one does so (eg. we haveseparate the descriptive theory, “Behavioural Portfolio Theory” that describes well how people invest: separate buckets for retirement, projects and playing on the stock exchange). As long as this cognitive dissonance exists, bankers and investors alike will be confused.

Maslowian Portfolio Theory (henceforth MaPT)provides a compelling answer: “people are right: one needs different buckets for separate investment goals”. MaPT starts from the the paradigm that investment are not necessarily a goal in themselves, they rather serve to support other goals: the real life-goals such as retiring, send your kid to school, leave a donation to a shelter.

That is the only tricky part. Once we agree with that, the rest is easy. Human needs are already described by Maslow in 1942. While some amendements are proposed to that theory the main point that “needs are separate in nature and that when one of the needs is not addressed it will get mental priority and will make us feel bad” still stands strong. From this automatically follows that the only safe wayt to help people achieve a satisfiable life is to make safe buckets for each and every big life-goal.

As a bonus we get that each and every of those goals sets concrete parameters that can be used to optimize the investment portfolio (eg. we need a sum of X in Y years). This takes the large amount af fiddling out of the equation too!

Well, it makes life a little more complicated for an investment advisor. focussed 1. Today they are all product-focussed. They need tfocussed o start speaking the language of the customer. 2. Today they are focussed on transaction stories like “invest in USA debt” and then a few months later have another story, inciting the investor to switch assets and so they can earn a transaction fee. So, advisors rather should take and advice fee. That is a little more complicated to explain. 3. They need a way to follow up these investments. That is not available yet.

MaPT makes things more difficult and almost postulates a complex and personalised follow up of the investment portfolio. So, it will be necessary to build such a software.

Maslowian Portfolio Theory in de Tijd

2012-02-15T14:55:00Z

The Belgian national financial newspaper, De Tijd, published on 3 Feb 2012 an interview with me in which the essence of Maslowian Portfolio Theory are explained.

The interview is by Pierre Huylebroek and is in Flemish only.

Donwload the page in PDF format

Thinking Coherently for Everyone

2012-02-14T15:09:00Z

The core idea of Maslowian Portfolio Theory explained very short and simple.

Much research has been devoted to a better understanding of financial markets. Much less studies are available about how to adapt an investment portfolio to the needs of a person.

Actually, Markovitz' mean-variance criterion (formulated in 1952) is still the dominant line of thinking. This paradigm where each investor has one investment portfolio that should cater for all investment goals, was also the base of recent legislation such as MiFID, FINRA and to some extend UCITS IV.

All those regulations use the concept that one investor has one risk profile and that volatility captures the concept “risk”. Doing so, they mostly ignore the different time horizons, importance and relevance of different investment goals. But worse, they have to rely on magical thinking and rule of thumb to match this mysterious risk profile to investments. When using a target-based approach it becomes possible to make reasonable assumptions about the different sub-portfolios and this should lead to more robust results.

This PhD puts financial investment into perspective: the investor’s perspective. The result is a new normative portfolio theory which confirms Behavioural Portfolio Theory, draws attention to the importance of asset-liability matching, and offers a natural framework for investor-adviser dialogue and mathematical portfolio optimization.

In this system investment goals, not investor psychology, drive investment advice; “risk” depends on the goal (often inflation-linked), and may be different in each sub-portfolio. Hence the ruling paradigm in which each investor has a single risk profile can be a misleading and dangerous simplification.

(Yes it took me more than 500 pages in the book with the same name)

Thinking Coherently for Everyone

2011-10-28T22:56:00Z

In this very short paper, I try to summarize the importance of coherent risk measures using clear examples.

The paper should be readable for everyone, and more in particular should be a pleasure to read for people who do not like mathematics that much … and btw, it is only four pages long!

Download

Thinking Coherently

2011-10-26T12:39:00Z

Niels Bohr once said that if all humanity would be lost and we were to pass only one sentence to the next generation that we should pass the information that matter is composed of atoms. Now humanity has built up almost hundred years more history, and with that hindsight I would prefer another sentence to be passed on. I would say “When thinking about risk, do it in a mathematically coherent way, that is only use convex risk measures that are related to the downside of the profit distribution”. I believe that our ability to start thinking coherently about risk will determine whether or not the capitalist system will survive and whether or not we leave a free world to our children or not.

Since 1638 expected value is was used as a measure for risk, during last century utility has been linked to risk and various risk measure have been proposed such as volatility. Back in 1997, Philippe Artzner and Freddy Delbaen and Jean-Marc Eber and David Heath, published a paper ‘Thinking Coherently’. With financial risks in mind, but with a deep and universal approach, they proposed four axioms that risk measures should satisfy in order to be “coherent”. To be honest, there might be more coherent sets of axioms, just as there are many coherent alternatives for Euclid’s axioms. Just as these alternatives describe another reality than the geometry on the plane (for example geometry on a sphere), it is also in finance possible to find other sets of axioms that are coherent. What is coherence in finance?

Different people might define coherence different, but few people would disagree that a coherent risk measure should satisfy the following conditions:

is an investment X has always worse outcomes than Y (in every observation), then X is more risky—in other words, a risk measure should be related to the downside of the investments (or the left tail of the return distribution).
If we invest more, our risk increases—Artzner asks a direct connection: twice the investment, then twice the risk.
If we diversify, the risk must decrease.
If we hedge our position, then the risk decreases with that amount—Artzner asks that if we add an amount of cash, that the risk should be decrease with the same amount.

The above statements seem to be acceptable, but are by far not the only possible way to define coherence. This definition of coherence seems to be fine for the investor that cannot influence prices, is not allowed to think in terms such as “once I default it does not matter any more”. The investor in Artzner’s approach cannot influence price and while an individual would not care if he has twice or ten times a debt that he can never repay, for the society it does matter. For example, once a bank defaults its shareholders lost their investment, and that’s it. They might not care if the bank defaulted with a net liability of one dollar or one billion dollar. The society that has to cope with the fallout (maybe a bailout) should care.

The most prominent issue for internal coherence might be the fact that a risk measure has only one minimum. Otherwise behaviour is erratic and has nothing to do with risk. For example, we would expect that if a portfolio is diversified more and more that the risk decreases more and more. Mathematicians would say that the risk measure needs to be “coherent”. This property ensures that We should only find one optimum, when we find local optima this means that our risk measure is internally incoherent and cannot be used to optimize portfolios.

Replace Text in Multiple Files

2011-09-01T22:27:00Z

Did you ever change your mind and want to change the name of one variable in your complete code base? Or change a name in all documents? You want to change one string with another in all files in a certain directory.

Linux gives you the right tools. There are multiple options.

Assume that in all files with the extention cpp (that are in the directory PATH/TO/DIRECTORY) we want to change the string OLDSTRING with NEWSTRING

Pipeing from find to perl

or if you want to make a backup before changing the files (and starting from Perl):

or (without making a backup and starting from Perl)

Example (exchange in all files):

While Perl would be in most repositories and is most probably installed, some machines with minimal installations (such as servers) might not have Perl and it might not be advisable to tinker with them. One fall-back solution would be to use the good old sed;

Probably it is the easiest to create a small file (eg. ch_all.sh) with the following content:

\#!/bin/bash
     for fl in *.cpp; do  #comment
     mv $fl $fl.old
     sed 's/OLDSTRING/NEWSTRING/g' $fl.old > $fl
     rm -f $fl.old
     done

Of course you should not forget to make the file executable and execute it:

Pensions Systems in Ireland and Poland, a quick comparison

2011-07-14T21:06:00Z

When Poland in 1999 reformed its pension system it displayed –despite some important mistakes that we will elaborate further– not only a courageous vision, but created a system that is inherently healthy and was an important investment in all the future generations. Pensions were to be taken from the current balance of the country and to be held on private accounts. That would free the state from an impossible situation at periods when the relative income for the state would be weak, providing critical strength in such difficult moments.

There is one generation that pays the cost. The generation that is working now funds the state pension system that is financing pensions not from its savings but from the current income. That generation will not only fund the old system but also save for its own pension. On top of that, this generation, should not expect anything from the old pension system when they retire. I believe that we all can be proud to pay this cost and free future generations from this burden. Besides –as Mr. Rostkowski rightfully notes– that will provide Poland a competitive advantage in the decades to come when other countries –such as the Czech Republic– fill face the bankruptcy of their state pension system and will have to make the switch in a later generation.

Unfortunately the difficult moments –that the new system would protect us from– came before the change was completed. The so needed reform of Polish public finance that made such a good start came to a grinding halt under the PIS rule and the economic boom of 2003–2007 was not used to make these important changes. Despite this Poland did well during the Global Meltdown and is still “first in class” in Europe. But during the crisis the public debt rose to almost the critical limit of 55% of GDP. Above that limit, Poland will face serious austerity measures from Europe, that seeks above all financial stability.

The government needed to act fast and strong at the eve of 2010. The state pensions system was a big chunk of the budget and that was an easy target: in one move the public debt could be reduced to a safe 47% of GDP. They decided to reduce the contribution to the private OFEs from 7.3% to 2.3%, and use the difference –5%– to fund the bankrupt state system. Unfortunately, the government made one important mistake. In stead of explaining that this is evil and that it is temporarily it decided to lie and pretend that the move was good and make the move permanent. The day that it was announced –December 2010–, Mr. Rostkowski explained that it would not change anything: “If we would give the money to the OFE, then they would buy bonds and so finance the public debt to pay for the state pension system. This is just a short-cut.” Once can of course ask why a government would introduce such change if it would not change anything. In any case, Mr Rostkowski is right from the point of view of the balance of the state, and yes, it has the advantage that Poland will not show 55% of debt, but 47% or so and hence pay millions less on interests (not only a lower outstanding amount to pay on but also a lower interest rate because a debt/GDP ratio of 47% looks better than 55%!). Unfortunately nothing can be further removed from the truth than this statement when your point of view is taken! From your point of view what happens is that the monies that were destined to be invested on your behalf on your account in order to contribute to your pension … well, they are now used to pay the pensions of the people that are now living from the state pensions system. The cash is gone, and there is nothing real –only virtual money, similar to Monopoly money– it leaves us with nothing more than a claim to the generation of your children and hope that they do display the same generosity as we do now.

It is sad that a system that proved not to function is allowed –as a vampire– to suck the blood of an inherently healthy system and by doing so crippling that healthy system. But this might be just what we need for a while (it could be the least of two evils!). However it has to be noted that the fact that this is hidden in a mist of half-truths does not help anyone.

Most reactions in Poland focussed on the risk that this Rostkowksi-reform would dry up capital acquisition of private companies (OFEs invest roughly 40% in equities) and hence hinder economic growth seems to me a true but very limited problem. Poland is one of the stronger economies and it has a great future. The fact that the economic growth will continue to be strong should support the stock exchange and while the growth might be slower, I believe that companies will still be able to find funding and support economic growth. The main problem is the one described above: presenting the vampire as the good one and crippling the inherently healthy pension system on the long run.

But when I speak of the reform of 1999 as an “inherently healthy system”, then I refer strictly to the essence of it as described above: funding of a private account and paying pensions from savings (not from current income). Many things –while historically forgivable to a limited extend– are an outright crime to the taxpayer. Let’s run through a few examples.

There are two major problems: the system feeds on a distorted view on how free market competition can be healthy (and thereby limit the potential to tailor investments to the investor’s need) and second it is expensive … very expensive, because it is designed in mistrust of the efficient asset management industry.

To illustrate these problems, we will use a hypothetical example of Mr Berczyk K., who earns 1000 PLN a month and we assume an allocation of 7.3% on top of that to the private pension fund. So, if you earn for example 4500 PLN just multiply all the numbers by 4.5 to find your figures.

Let’s start by choosing a reference. I work in Ireland now and co-run a very small pension fund of only 20 (twenty!) employees. The pensions fund is a simple securities account with a large provider and all costs are paid by the employer (so separate from the pensioner’s savings). This small fund gives me enough purchasing power to bargain a 0% entrance fee and a 0.5% management fee on a global equity fund (while of course having a wide range of –free to choose!!– funds for all employees in the system! Let’s name the Irish investor Paddy and let’s make abstraction of the huge differences in salaries between Ireland and Poland and hence assume that he earns exactly the same as Berczyk. The polish investor will not even get the benefits of global diversification and –what is infinitely worse– not the benefit of free choice of strategic asset allocation (but more about that later), on top of that he pays 3.5% entrance fee and an average management fee of 1.38% (2009 figures, the 2010 numbers should be lower).

We further assume simply a stable risk premium of 3% above 3% long term interest rate and assume that both Mr Berczyk K. and Mr. Paddy O. (from my small pension fund) are 20 years old and retire at 65 years old. If we only look at the difference in costs and assume an average return of 5% per month for both of them, then Mr Paddy will retire with 164 thousand euro and Mr Berczyk with 138 thousand, a dazzling 20% difference! But, Mr Paddy is free to choose his own strategic asset allocation. Assume now that he simply chooses for equities till he’s 55 years old and the last ten years uses fixed income investments, while Mr Berczyk will always have a 50/50 investment over bonds and equities. In that case we expect Paddy to retire with 121 thousand euro in his pocket and Berczyk with 90 thousand, a difference of almost 35%. But that is not the most important here. Paddy can as he get closer to retirement change to a safer strategic asset allocation and completely eliminate the risk of a large market drop-off in the last years (when there is no chance that his investments will recover till retirement, because the time is too short), while Berczyk cannot. If the markets drop off a cliff the month before Berczyk' retirement (assume 30%), then Berczyk will have to live with about 50% of Paddy!

Sure, a 30% drop-off is extreme on a diversified portfolio, but of you have been working 45 years for those savings even a 5% drop-off will not make Berczyk smile. It is not too difficult to eliminate most of the cost inefficiencies, but the worst of all inefficiencies is the fact that Berczyk cannot change his strategic asset allocation. And that can be changed without any effort at all: simply delete the ridiculous law that forces OFEs to copy each other or face penalties and rather encourage that they offer a 3 or 4 different risk profiles for the investor to choose from. The actual system where a pension fund that differs in performance risks to have to pay into the fund is a very costly one (the owners of the pension funds are very cautious after the many mood changes of many governments, but especially the last will make a PTE their worst investment … that is probably the most important reason of the high management fee). That system is of a demagogic importance, but intelligent people as Mr Rostkowski and Mr Tusk should be able to look through that and do what is best for the Polish Citizens: change those rules and explain why.

On top of that, if we assume –in the worst scenario– the reduced payments of to the private system, Berczyk will have only 20K, that is six time less than his twin brother in Ireland. For the rest he would have to rely on the charity of the working generation in the future to continue to add something to his pension from the running balance. That new generation could then well be in an impossible situation because of the extra pressure that Mr Roskowski added to the current national balance (combined with some future problems).

Complicated Investment Products

2011-06-22T10:19:00Z

I believe that by now I have accumulated some experience after 15 building business and recently finishing my PhD. However some things continue to surprise me. The recent crackdown on so called complicated products—on both sides of the Atlantic Ocean. I still have difficulties to understand what it is about?

Would you refuse laser-therapy to a patient because he or she is not able to understand the laws of quantum optics? Would you refuse someone to buy a care with an airbag because he or she is unable to understand its IC circuits? Wouldn’t that be extremely unfair?

So would it be fair to refuse to someone the best fitted investment strategy because “it is complicted”?

What would have happened to the car industry if hundred years ago a law would forbit to make “complicated cars”? What would have happened to life expectation if regulaters would crack down on complicated medical treatments? Why would anyone do that?

But in the investments industry it is something that seems to be loved by the media … or that is at least the only reasonable explanation that I can think of. Why else would regulators do so? For the sake of control? The fun of discriminating the “stupid masses”?

It gets even more surrealistic when the regulators get down to details. For example the Belgian regulator published on June 20th a press release calling for a voluntary stop on “unnecessary complicated products”. They even have the wishdom to define what is unnecssary complicated. Any investment product that does not satisfy all of the four following conditions is considered to be “unnecessary complicated”.

The underlying value is sufficiently accessible — This is stangily formulated it can mean that it is liquid, available for sale for everyone (but then … why would you need an investment fund), or it could refer to the existance of a market price. In all of the three possibilities this seems absurd to me. In each of the possible interpretations I would argue that if such investment is usefull that then one exactly needs an investment fund to make it safely accessible for investorsl
The investment stategy is not too complicated — This is so vague that I cannot comment. But —see further— this does not withold the FSMA to list financial institutions that will not abide by these rules.
The retun is not determined by more than three mechanisms. — Again vagueness and unclarity reigns. Would an bond portfolio qualify for example? Well the return is determined by at least the following mechanisms: (1) the mechanism determining the price of each bond in function of the interest rate curve, (2) the mechanism that make the interest rate curve move in function of utterly complicated interactions in international financial markets, (3) the mechanism that the portfolio manager uses in order to select the bonds, (4) the mechanism links the return of the bonds to the creditworthiness of the emittent … Well, the limit is exceeded: a bond portfolio is “unnecessary complicated”.
there is complete transparancy about all costs, credit risk and market value — This seems on the first sight to make sense, isn’t it? But let’s have a look at a simple structured product that offers capital protection and increase in function of an index. This is done by buying a long call one the account of the fund. How much cost is in that long call? The fund cannot know this, actually even the counterparty who is selling the call cannot know the actual cost in advance. The seller has mathematical models that make many assumptions (and when reality will be different than the assumptions it will make more or less profit or evel loss) and it adds to that an error margin and a profit margin. These margins re of course unknown by the fund, and each option seller has other models, parameters and margins. It is clearly impossible to know “all costs”. Even a simple bond or equity fund can never predict the exact amount of transactions and hence the transaction costs, even the transaction costs per transaction cannot be predicted exactly. This statement is hence misleading, and an oversimplifiaction. As if someone really would believe that “all costs” can be known in advance.

Do you see any logic behind these regulations, are taxpayers and investors paying regulators … to get discriminated and get worse investment products? Would we not be better of to focus on what really matters? A coherent downside risk measure?

Sure, an understanding of coherent risk measures among law makers and regulators would be a leap ahead, but honestly … there is another story too. From marketing we know that in absence of a clear reason to connect to a brand customers are not that loyal and can be “manipulated” by price, peer pressure, … and novelty. This search for novelty is out of control among producers of shampoo, toothpaste … and investment product providers. This indeed resulted in pruducts that are “unnecessary” complex. We’re talking now about the products where an investor buys basically an OTC option that is highly tailor made for the intermediar. These products would indeed sport things like a bespoke selection of stocks, with special features such as some dropping out or entering depending on this going up or that going down, etc. etc. Indeed for me these products are complicated to understand … even more so because typically sales people –who have little experience with pricing of options– do not really understand them.

It is not the first time that a private banker contacts me with an exclusive product offer “full upside potential AND capital protection”. Investments should in the first place encourage people to save in long term. To get it right it has to be simple enough so that something sensible can be said about the evolution of the return distribution over time so that one can link it with his/her own portfolio and link it to personal goals (and typically one would expect a monotonous link with GDP).

A set of 5 to 10 building blocks are largely sufficient to cater for all needs, fears, wishes and plans in each individual’s portfolio.

Target Oriented Investment Advice

2011-03-19T20:36:00Z

This paper aims to present concrete recommendations to individual investors and in- vestment advisers about what portfolios should be held by private persons and how to construct these portfolios. Starting from the theoretical foundations that are based on the “Hierarchy of Human Needs” (as proposed by A.H. Maslow in 1943), this paper suggests to segregate portfolios into multiple sub-portfolios that each cater for a specific need. This postulation is then integrated with other key observations into a comprehensive, positive and normative portfolio theory. The result is a complete framework for personal financial decision making that is natural and helpful for both advisers and investors, while it integrates financial investments in an optimization of overall well being.

Download Flyer

Polish Politics

2011-03-06T10:19:00Z

I spent almost a decade in Poland, and live the last two years in Dublin, one of Europe’s financial centres. This allows me almost the viewpoint from an outsider on Poland. Poland is both a cherished example in Europe and an economy that worries its European partners.
Poland was the only European country that did not have recession, and the growth in 2010 is estimated at 3.5% and this year it should be higher (3.7%). Poland can expect to earn well from privatisations and money from the private pension system is diverted to reduce costs for the state, the football games of 2012 will fuel investments and lead to improved infrastructure. And as the icing on the cake, there are of course the European Subsidies. Also the political situation is perceived generally well. The leading party has still a majority of about 10% in all polls, indicating a stable situation around a government that is considered as responsible and not totally populist and reasonable. The improved relations with Germany and Russia were a welcome success for the government and it strengthens their reputation when they take the leadership of the European Presidency later this year.

But also beyond this success story there are many aspects that make this success a rather weak wrapping. The successes in international politics are overshadowed with the problems in Belarus for example. The government seems to be easily distracted in the populist game making big fuss about a public holiday or about semi-legal drugs. Such displays can be successful in the bidding for a certain group of voters but don’t make a good impression in international financiers that are lending money to us (and their appetite for buying the debt sets the interest rate and hence the cost for the country). Indeed where the Tusk government could hide after the veto fever of the previous president it doesn’t seem to be able to get the focus right there where it should be speeding up the needed reforms in public finances, administration and even the labour market.

For example, we all know that our police and farmers are not the big earners, but their privileges such as tax exemptions and early retirement draw attention. It’s just a matter of bad public marketing: increase the salaries for police and withdraw their privileges and we avoid the bad press. I know it’s unfair, but we are still an “ex communist country”, and such privileges are seen as “communist era privileges”. And that is not the label that you want when you want to lend money.

Some reforms such as the pension system where the money of working people is diverted to the state (the OFE of the private pension would only get 2.5% in stead of 7.3%) and the difference goes to the state. Sure, we know it goes actually to ZUS, but their the money will be consumed in stead of invested, and that will create disastrous consequences at the long run. This move makes the outlook on the long run rather grim. This is a move in the wrong direction, but personally I believe that on the short term it can be the worst of two evils.
Indeed the alternative was to present an even bigger public debt (also this is not advantageous for the tax payer), but more importantly Poland is struggling to keep the public debt below the 50% of GDP level. Above that level, Poland would have to face harsh austerity measures. But in international financial circles it tastes bitter to hear that the so necessary reform of the pension system is put a step back only a few weeks after Poland got an agreement to keep the cost of the pension reform out of the definition of public debt even till the end of the reform in 2060.

Besides, the deficit is now 7.9% and that is four times higher since Mr. Tusk took office. Sure, we know that that this can be attributed to the lack of reforms during the boom period under the previous government. Now that the global economy pulls us down, it is a lot more difficult. This is one of the many reasons why the pension reform on the long run should continue and why it is dangerous for taxpayers and pensioners to keep it in the balance of the state

Also the Polish infrastructure, mainly highways and and rail-roads don’t have the best reputations. Announcing delays has to be avoided. The energy production also needs serious plans to get rid of the polluting fossil fuel

Finally, it might be advisable to keep the elections away from the European Presidency, because that is a unique and important opportunity to show Poland to the rest of the world as modern nation that is on a right course to the future

No More Bailouts

2011-03-01T11:06:00Z

“The tax payer has been funding the banks more than enough! No more money for banks! Make the rich pay!” It’s en euphemism to state that similar points of view are the common denominator in the press that reports on the dramas since the Global Meltdown in 2008.
There are a few problems with these popular statements though. There is a misconception and there is a complicated truth about complex, interdependent economic dynamics.

First, the simple misconception.

After a month of intensive reporting about the bailout in Ireland, I read in a newspaper “How could Lenihan [finance minister and official laughing stock of Ireland] take money from the Pension Reserve Fund? No money can be withdrawn now”. When reading the press reporting about the media about the “bailouts”, it is easy to understand where this misconception comes from. Media reported about the bailouts as if it was a donation (and continue to do so). In reality the bailouts are loans and/or share purchases.
Loans that have to be paid back or shares that can be sold later.
The Troubled Assets Relief Program (TARP - that was installed in the USA in October 2008 for example has for 2/3rd been paid back and on those assets paid back there is a 15% profit for the taxpayer (or almost 10% profit on the whole portfolio; while still about 100 billion has to be paid back). This is to date a profit of almost 30 billion USD for the USA taxpayer.
Of course the outstanding 100 bln is still at risk (if it will be paid back there is more profit, but the involved companies can still go bust and not pay back).

Second, the complex economic dynamics.

This subject is so complicated that I hardly can blame the press to do an effort to explain this. True, it is more the subject for a PhD than a one-pager blog, but if one sticks to the basics it is not too complicated.
First we have to distinguish the bailouts for countries and those for specific companies (such as banks).
A bank operates to some extend similar to a hedge fund in that sense that it leverages many times its assets and by that leveraging it creates cash that is used by other industries. This is essential for economic growth. In the wake of 2008 we have seen that the major problem for everyone was to get loans (because the banks had problems). This has lead to economic depression (except for Poland that is) and made other markets tumble. We have to underline that in 2008 there was only one bank that went belly up, without the TARP it could have been much worse. The major problem of a bank going bankrupt is the domino effect that it creates: it will pull other banks down and all banks will have to be more restrictive and so they impede the economy.
The story for countries is also complex, but also here the alternative to a bailout is bankruptcy. When a country defaults on its debt it can as a sovereign choose how to get out of the situation. The biggest problem is to gain confidence of the investors in order to be able to emit new bonds. So the state will typically devaluate its currency and then pay back a part of the value of the bonds in that devaluated currency (for example pay only the face value and no interest). The years to follow the state will pay very high interest rates because of the weakened confidence, the interest rates in the country will be high (impeding the economic growth), inflation will be high, etc.
The main problem is that after a default a state will not have easy access to funding. It will have to cut on many expenses, probably more so than in a bailout scenario! So people that are protesting against the bailouts should realize that the alternatives are not necessarily better. Indeed the cost cuttings that accompany a bailout (because remember, it’s no charity it’s a loan that has to be paid back – so the ones that provide the money (not “give”) want the country to improve its finances in order to be able to pay back the investors)

By allowing one country to default on its debt, Europe creates a precedent and as that will trigger a loss of confidence in Europe, the cost of funding will dramatically increase for every taxpayer in Europe.

All that is true, but it does not exclude that a bailout is socially a reasonable solution. The alternatives are also harsh for all layers of the society.
The only good solution for the future is to avoid such situations. This is not something that a invisible hand can do: real regulation of the banking system (that goes beyond demagogic rhetoric) is a first step. Guidelines of tax structuring, leveraging, balance, debt ratios, debt structuring, debt currency, with real risk parameters such as CVaR to be used (not flawed –but popular– measures such as volatility or VaR!

Lessons from 2009 for Poland

2010-12-30T20:02:00Z

Poland was the ONLY country in the EU to avoid recession in 2009. In 2010 we expect again the record level in Europe (around 2% GDP growth). How was such an economic top performance possible?

Probably the most important aspect was that Poland had a relative low debt ratio (about 50% of GDP) and that this debt was mainly in the home currency. With hindsight one might say that the lack of economic stimulating initiatives and the resulting slow growth and low debt under the reign of the Kaczynski-brothers wasn’t the worst option.

The truth is more complex. In fact the country was well prepared in the nineties by people such as Leszek Balcerowicz (by building up important foreign reserves, removing price controls, eliminating many subsidies, imposing a strict monetary and budgetary regime, etc.). The economic expansion of 2003-08 gave PIS an easy ride and the fact that this growth was not used to create a healthier budget balance is a unique lost opportunity. Improving that balance becomes now in the economic downturn very important and it is of course much harder to do it now. Many more issues were not addressed by the PIS rulers.

The credit line of the IMF that was allocated in May 2009 (20.6Bln USD) did not only provide added creditworthiness but it was also seen as a confirmation of the IMF’s belief in the countries ability to address the crisis and respond appropriately.

Other important factors were the relatively strong internal market, the low level of debt of individuals (that avoided a real estate crash for example), the responsiveness of the the zloty (sharp depreciation, but then again a steady climb indicates that the possibility for the PLN to float did not have much real value but psychologically it helped), the external imbalance was modest to ok, the dynamic job market allowed salaries to react, and of course the enormous EU cohesion funds.

Poland became the largest recipient in absolute numbers: about 3.3% of the GDP for the years to come. We must use these funds to create structural economic independence for the country, create a climate of competence and competitiveness, and attract more structural Foreign Direct Investments (FDIs). To achieve that, there are a few areas in which we will have to improve.

Corporate taxes are too high in order to compete with the Czech Republic, Bulgaria or Ireland. Although just decreasing taxes is not a good idea, because this becomes pro-cyclical. This means that lower taxes will lower the income of the government in an economic downturn, exactly when it needs as stable base. This means that the government should seek to shift income base from companies to private persons (why keep the special advantages for farmers?), real estate, certain forms of ownership, alcohol and cigarettes, natural resources, emissions, etc. To keep the internal market strong, a low VAT policy would be beneficial. Trustworthiness. A government should strive to continuity in economical and fiscal policies. An investor does not choose Poland because one government lowers taxes. Only when he believes that for the decades to come this policy will be sustained he will consider Poland. Simplify and modernize laws. Why would each district be allowed to have a different interpretation of the utterly complicated tax laws? What a waste of resources in both public and private sector, what a wrong focus for the investor! Would it not make sense to strive for transparent legislation where no interpretation is possible. Improve our infrastructure such as roads (and maybe consider something stronger than asphalt), fast trains and especially environmental friendly and efficient public transport such as trams. Introduce the Euro fast. Insecurity about the exchange rate is a big damper on FDIs.

In short, we should now start to prepare for the day that the money flow to Poland will stop! We should remain modest and don’t spend more than that we have (i.e. keep the government debt low). And of course keep the associated risks low, so have it in PLN or have it hedged to PLN).

The Seven Myths of Investing

2010-12-01T21:14:00Z

In our economic system it is essential to take responsibility for your own financial well being. Where under certain protective economic systems the state will care for the individual to a large extend, these systems tend to come with loads of disadvantages. We’re all happy to live in a free world … but this freedom also includes the freedom to die poor.

The answer is to use investments. After 2008 we all know that this is not so easy, many people are still recovering form serious losses and feel uncomfortable to any risk. Only the safest thing such as the savings account or term deposit can be considered. But by doing so unintentionally one risks to earn less interest than the inflation … and therefore erode one’s capital.

While this might be a good prospect compared to incurring huge losses on the stock exchange this strategy is bound to be suboptimal if your time horizon is longer than a few years. The only answer for a happy and sorrow-free retirement is investing.

Investing is for the novice and experienced person full of myths and mysteries. We’ll look at some of those myths now:

I need to be fast, the slow are lunch for the fast. That’s what online broker platforms would like you to believe, but nothing is less true. Research (e.g. Terrance Odean and others) persistently finds the opposite: trading more erodes your investments (even without taking the trading costs into account). There is something with high frequency trading, but then we’re talking about sub-millisecond trading. Lesson: think about the investment goal and select an investment profile that suits the investment horizon and only look at your portfolio every year or so.
My banker knows! How many times did you raise the questions “What should I invest in now?” That’s not an investor’s question it’s a gambler’s question. And what are the odds that your advisor would keep his old job if he would really be able to tell? Lesson: Ask the right questions! An investor starts from a goal and investment horizon, then selects a strategic portfolio composition.
Investing is only for the smart. Financial press would love you to believe that, but see point 1: generally people that believe themselves to be smart are overconfident and trade too much and hence have poor performance. You need a basic knowledge about investments and their risks, but don’t need to read every day the financial newspaper. Lesson: don’t get lost in short term news overload, but focus on the long term goals and fundamental risks.
I’m too unique, I need a special (expensive) structured thing. We’re indeed quite unique in many aspects. But investing should be the answer for a real need, and those needs tend to be less unique as you would believe. Harry Maslow groups all human needs in a handful of layers. Investments are a tool to cater for those needs. Lesson: you need a few portfolios with each a clear goal (retirement, schooling children, and special projects).
I need a lot of money to start investing. While it is true that the best way to collect a small capital on the stock exchange is to start with a large one, there do exist efficient structures that allow one to invest for very small amounts. Investment funds, Exchange Trades Funds, and some index notes will provide a well diversified portfolio for amounts starting as small as a 100 Zl. Lesson: use simple structures and make sure you get the diversification that you need. Consider a systematic savings program.
I’m too young to think about my retirement, I still have time left. Consider this: when you save every month from your 20th till your 60th year of life 100 Zl a month and assuming an interest rate of 5% per year this will yield 148'956Zl at retirement. If you only start at your 40th, you will not have to invest 100, but 392 Zl per month in order to get the same result. Lesson: the early bird gets the worm.
I need specialized press. Not any more! I plan to bring more concrete advice, look for the the next Blogs in this section!

High Frequency Trading

2010-10-23T20:35:00Z

In the wake of their public face off in 2008 where during 6 hours and 45 minutes they were unable to consolidate any trades, the London Stock Exchange (LSE) announced that will upgrade it’s .NET TradeElect (Windows based) systems to the (MilleniumIT Linux) based platform. Following hereby the New York Sock Exchange that made this move in 2007.

By doing so the LSE, one of the few world-top stock exchanges, hopes to reduce their transaction speed to 0.4 milliseconds. This is more than 6 times better than the (in fact rarely observed) speed of 2.7 millisecond in their Windows based solution.

This should not surprise us: Linux gained a market share of 91% in the top-500 supercomputers (up from about 0% in 1998), it powers the top 15 without exception. It is just part of the success story of Free and Open Source Software (FOSS), where propriety systems lost the battle long ago in embeded systems, mobile devices (although the Redmond based company fights back with an OS and … of course legal claims)

I might not just have been speed that drove LSE to Linux, Linux is also a lot cheaper than Windows on such systems. Probably the most important argument is that now LSE is in the driving seat and that it is not held hostage by a network of monopolies.

According to the IBS Journal David Lester, Chief Information and Technology Officer at LSE, said that compared to the annual cost of $65 million for TradElect, MillenniumIT was a bargain at a purchase price of $30 million. Sure that is not the whole picture, but thehe LSE predicts that using Linux will give them an annual cost saving of at least £10 million (EUR 11.2 million). He adds: “The new technology is a lot lighter, nimbler and easier to install”.

But is that the whole story?

On the other hand we have fore example the May 6 Flash-Crash, this aberration in stock behaviour has cost some investors many millions, whereas for others it might have been the best day of the year. As a matter of fact, in my time zone it was about 8:30 pm and I was where I should be at such moments in front of my Bloomberg terminal ready to pull the trigger and make or loose a many many millions. Fortunately my blood was chilled enough and helped by the approach of my employer I avoided slipping our short (hedging) traders in the black hole created for a few seconds.

Even with hindsight one can only see that the markets on May 6^th 2010 were very nervous about the possibility of Greece defaulting on its outstanding 210 billion euro debt, and over hours after closure in Europe markets declined a few percent, but then suddenly the debt dived 600 points in a matter of minutes. The stock exchanges followed that dive-boom scenario, with of course the U.S. Stock exchange being open the most visibly impacted. The dive erased about $862 billion from U.S. Equities alone! It was the largest intraday drop ever.

Regulators in the U.S. anounced weeks later that they wer investigating 6 possible areas, but it seems that HFTs have during the build up of the crash put thousands of trades per second in the stock exchange, only to cancel them within milliseconds.

Hedge Funds and day trading are indeed so passe, it’s high frequency traders that are the new rebels of Wall Street. High Frequency traders are not interested in the fundamental value of a stock, but use mathematical algorithms to profit from other players’s actions. Some of them employ hundreds PhDs and use computers to generate thousands of trades per second, these trades are based on the trends of the last milliseconds and have a millisecond delay and are more often cancelled after a few microseconds than they are really executed.

There are basically two (known to me) ways of operating for high frequency traders. A first model is operating as market maker, such companies have permanently buy and sell trades in the market and take no directional bet. Another model is looking for local and temporally mis-pricings in stocks or … even inefficiencies in the systems of the stock exchange or between the different exchanges. But in all cases the High Frequency Trader is only interested in making a quick kill and moving on.

A simple expample would be a pairs-trade, where for example bad news pushes a whole sector down. HFTs will capture the first movers and then take positions of on the others in order to profit from those comming down also. Or a step further one could try to forecast a certain companies short term moves by monitoring a sector index, it’s raw materials, interest rates, etc. A second example is simply monitoring the bid and ask spreads and acting then they come too high and profit from conversion. But, do not try this at home! The active HFTs do not simply employ computers to look for patterns, basically they are doing what Keynes in in beauty contest parable stiplated. They employ computers to look for other computers that look for patterns.

Some researchers have found many cases of thousands of trades that were put into the stock exchange but were cancelled within milliseconds. This raises the suspicion of intentionally influencing the stock exchange’s systems. This is referred to as “quote stuffing"e; and is obviously on the limit of what is legally possible. Or is it just past that fine line? In any case regulators are investigating and considering what to do?

A simple model would be to watch for such large orders (eventually on an aggregated base) and when spotted take the same direction very fast and before the order is fully through the market reverse your position completely and therefore buying low and selling high in the case of one large buy order. But the true algorithms are known by only a handful company owners and CIOs (chief investment officers, not chief information officers!), most people will only know their part of the patchwork that is the web spun by their company around the stock exchanges.

If the asset managers who build up for you a retirement capital are the farmers raising livestock for you, then the High Frequency Traders are the hunters that lay in ambush to kill one of those cows when it comes too close to their trap (or at least take today’s milk).

But we should ask ourselves is that the stock exchange that we want for our children? Is that the stock exchange that will play its role as a catalyst for the economy and provide oxygen to growing companies that produce and create welfare?

Clearly there is an issue here and the U.S. Securities and Exchanges Commission (SEC) is considering if and what action to take. They consider for example imposing a minimum period of time that trades have to remain active (probably something in the order of 20 to 100 milliseconds). The SEC is further considering to monitor more closely firm that trade more than 2 million trades a day, and to force HFTs to stay in the markets when the going gets rough.

Institutional investors are already spotted the problems and try to put up an armour against what must feel like parasites. One technique is breaking up large orders in small pieces so that you show to the HFTs only the top of the iceberg. But as long as the regulator do not interfere it’s the same exhausing battle as between anti-virus software maintainers and Botnet Masters.

Institutional players such as investment funds cannot longer simply use block trades and they have to use the same techniques as the HFTs in order to defend themselves. Also hedge funds that work to find patterns and trade on them have to watch for HFTs to jump before them when taking action (triggered by themselves showing their intentions), and probably shorten their holding periods as market anomalies that a few years ago would have lasted for days now will disapear in minutes.

But as long as there are no regulations about the minimal lifetime of quotes or simply minimal processing windows (for example the exchange will only process and match trades every 100 milliseconds) then the stock exchanges, being private companies, will continue to compete with each other in price and speed "e; paving the way for their own problems. Just as banks did in the run up to the Global Meltdown in the lack of serious regulation! This is one of those examples where the regulators can and should play an important role and where they can do more than window dressing and finger pointing.

Regulators have to rethink their strategy of the last decade where since the introduction of the ECNs in 1998 and the decimalisation in 2000, and the best-and-fastest execution rule in 2007 (all USA) they have been striving in making markets more efficient on a micro level.

And You? Do you still believe that you need on-line information and daily newspapers to manage your broker account?

How Safe Is Government Debt?

2010-10-04T21:08:00Z

Looking for safe investments? Government bonds!

But is that really so? And if not, what tools do we have to assess and compare the risks? Is there an issue with Greece, Ireland, Italy, Portugal and Spain (the “GIIPS countries”)?

First of all there are the rating agencies. They assign classifications represented by letters to all debtors. For example Standard and Poor’s uses letters A, B, C and D. AAA is the best, then AA+, AA, AA-, BBB+ etc. all the way down to D for companies that have defaulted. Anything from and above BBB- is called “Investment grade” and anything below BB+ is a “Junk bond”. Each rating agency uses different symbols and has its own methodology. There are two ratings per debtor: one on short term and one on long term. The rules to assign the letters are different for countries and companies and even depend on the sector in which the company is active. The rating gives a good indication of the probability of default.

Looking at the ratings we see that Greece is in troubles, but that most probably the rest of the GIIPS countries will be fine. So no reason to panic it seems.

Entity	Standard & Poors	Spread (in %)	CDS (in %)
Greece	BB+	7.93	6.84
Ireland	AA-	3.30	4.21
Italy	A+	1.06	1.73
Portugal	A-	2.74	3.46
Spain	AA	1.30	2.05
Poland	A-	3.56	1.19
Germany	AAA	0.00	0.32
USA	AAA	-0.53	0.38
Indonesia	BB+	4.76	1.29

Another way to judge the risk of a certain country is to study the “Spread”, the difference in interest rate that the debtor has to pay compared to a chosen reference. For European countries one generally compares to Germany, one of the world’s most creditworthy countries and a dominant debtor in Europe. The spread is hence a measure for the risk premium that investors demand in order to buy certain bonds.

Therefore the spread reflects the common opinion of all investors; it is a consequence of the trading going on between all players. Where rating agencies very seldom adapt their views, the spread changes intraday just as interest rates or stock prices.

In order to asses the risk of a bond it makes more sense to study the spread than the interest rate itself in order to eliminate the overall interest climate.

The spreads in the table already give us more reason to worry. Greece has a higher spread than Indonesia that defaulted a few years ago. Further we see that actually Ireland is assessed to be more risky than Italy and Portugal, whereas the Ratings gave us the opposite image. This can still be consistent, because a buyer of a bond will not only be interested in the probability of default, he will also be interested in how much he still can recuperate in case of default.

Also a Credit Default Swap (CDS) is an ideal instrument to assess the credit risk of a certain bond. If you have a bond, you can cover its default risk by buying a CDS, the seller will then pay you the face value of the bond in case it defaults (and you give him the paper, so he can recuperate whatever is left). The joint probability of default and amount to recuperate is most explicitly present. But there is another aspect that plays a key role.

You have your bond and CDS, but you still have a credit risk on the seller of the CDS. So, this operation only makes sense if the creditworthiness of the seller is better than that of your bond. In other words, the market of CDS providers is small. And those who do will have a portfolio of contracts. Hence a third element of the risk of the bond comes to the foreground: the systemic risk: when it goes wrong, how strong are the bonds correlated?

The CDS values in the table learn us that there is more matter of concern for Greece, Ireland, Portugal, and Spain than there is for Indonesia for example. And that is a reason to take precautions!

Indeed things can go wrong: companies can default, even Lehman Brothers or Enron were not immune. Also governments can stop paying or pay only part. Take for example Argentina. After the two wars the country could not gain the confidence of investors and it ended up paying high interest rates and had hyperinflation. The situation exploded in December 2001: the country had its currency devaluated and paid back partly in the devaluated currency.

None of the Euro countries has the option to devaluate its currency, so the income from taxes minus current expenses determine the limits.

That reflection allows us to look via a different frame at the same reality. Greece has a debt of 116% of its GDP, but we must not forget that this is only 118% of its annual revenue (total income from tax). If we compare that to the USA who has 53% debt-to-GDP but a stunning 358% debt-to-revenue ratio, we must conclude that all is still relatively under control.

Buying GIIPS debt is like jumping into a driving train: hopefully it works out, but when it goes wrong then it really hurts. The first default will trigger panic among inventors and the IIPS countries will see their cost of funding soar. One domino will make the others tumble. In other words let’s hope for the best, but when it goes wrong it won’t be easy to hide.

NOTE: Another interesting observation while comparing CDS and spread is that the EU countries have an interest rate advantage compared to Poland for example, but more about this in another paper about the advantages of the Euro.

Investment Risk and Risk Classes

1998-10-01T20:26:00Z

This paper aims to present a risk classification that can be used for all investment funds and that is close to what the investor perceives as “risk”. The method further tries to use as few arbitrary parameters as possible and the assumptions are limit to the maximal extend. We present one graphical method that could be used in a discussion with the investor, and a risk classification that is so simple that it can be used on written communication. Ideally the risk class should be printed on each fund information sheet, so that investors can compare all investments over all different distributors.
1998-10-01

NOTE: this is an old paper, though this paper is still interesting reading, please take the following into account: - Recent research points rather in the direction that a risk scale at the level of the investment itself might not make much sense. An investment is risky or not depending on the details of the investment goal! - That paper misses an extremely important link to other papers from that time and more concretely to the paper “Thinking Coherently” (Artzner, Delbaen, Eber and Heath — 1997). This paper erroneously selects Value at Risk (VaR) as a risk measure. VaR is not a coherent risk measure and should not be used (certainly not at the level of individual investments).

Donwload the paper in PDF format