How to augment lpsolve R optimization solution to run on a hadoop cluster?

Question

I am using R lpsolve package to optimize my transportation model. My code runs fine but it takes a lot of time to run as I have huge number of nodes and paths. I am planning to run my code over hadoop cluster.

Please guide me regarding changes that i need to make to my code. I think that running optimization over hadoop cluster might be impossible as we might end up with local minimums instead of the global minimum.

I search internet for terms like "lpsolve hadoop" but didn't get anything useful.

Please direct me to material or examples that i should look at. =====================================update 1====================================== The original problem that I had is here.

I simplified the problem further and the current problem that I am solving is as below.

The R code and the input data file that I have built using excel is attached. In real scenario, input data file will be generated using SQL and will be >30,000 rows in length.

enter image description here

My input excel file:

startlink   endlink link_dsc    lnk_type    cons_type   cost    equality_const  fpc_const   max_const
"source","-","-","0"    "vmi1","MM1","VMI","1"  source_to_VMI   supply  equality supply 0   100 null    null
"vmi1","MM1","VMI","0"  "cust1","MM1","SHIP_CUST_1d_AIR","1"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","1"  "cust1","MM1","SHIP_CUST_1d_AIR","2"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","2"  "cust1","MM1","SHIP_CUST_1d_AIR","3"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","3"  "cust1","MM1","SHIP_CUST_1d_AIR","4"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","4"  "cust1","MM1","SHIP_CUST_1d_AIR","5"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","5"  "cust1","MM1","SHIP_CUST_1d_AIR","6"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","6"  "cust1","MM1","SHIP_CUST_1d_AIR","7"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","7"  "cust1","MM1","SHIP_CUST_1d_AIR","8"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","8"  "cust1","MM1","SHIP_CUST_1d_AIR","9"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","9"  "cust1","MM1","SHIP_CUST_1d_AIR","10"   vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","0"  "vmi1","MM1","VMI","1"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi1","MM1","VMI","1"  "vmi1","MM1","VMI","2"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi1","MM1","VMI","2"  "vmi1","MM1","VMI","3"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi1","MM1","VMI","3"  "vmi1","MM1","VMI","4"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi1","MM1","VMI","4"  "vmi1","MM1","VMI","5"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi1","MM1","VMI","5"  "vmi1","MM1","VMI","6"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi1","MM1","VMI","6"  "vmi1","MM1","VMI","7"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi1","MM1","VMI","7"  "vmi1","MM1","VMI","8"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi1","MM1","VMI","8"  "vmi1","MM1","VMI","9"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi1","MM1","VMI","9"  "vmi1","MM1","VMI","10" vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi1","MM1","VMI","10" "SINK","-","-","100"    vmi_to_sink esacpe  null    100 null    null    null
"cust1","MM1","SHIP_CUST_1d_AIR","1"    "cust1","MM1","CUST","1"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_1d_AIR","2"    "cust1","MM1","CUST","2"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_1d_AIR","3"    "cust1","MM1","CUST","3"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_1d_AIR","4"    "cust1","MM1","CUST","4"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_1d_AIR","5"    "cust1","MM1","CUST","5"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_1d_AIR","6"    "cust1","MM1","CUST","6"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_1d_AIR","7"    "cust1","MM1","CUST","7"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_1d_AIR","8"    "cust1","MM1","CUST","8"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_1d_AIR","9"    "cust1","MM1","CUST","9"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_1d_AIR","10"   "cust1","MM1","CUST","10"   shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","CUST","4"    "SINK","-","-","100"    cust_to_sink    flow    demand  0   null    null    50
"vmi1","MM1","VMI","0"  "cust1","MM1","SHIP_CUST_2d_AIR","2"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","1"  "cust1","MM1","SHIP_CUST_2d_AIR","3"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","2"  "cust1","MM1","SHIP_CUST_2d_AIR","4"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","3"  "cust1","MM1","SHIP_CUST_2d_AIR","5"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","4"  "cust1","MM1","SHIP_CUST_2d_AIR","6"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","5"  "cust1","MM1","SHIP_CUST_2d_AIR","7"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","6"  "cust1","MM1","SHIP_CUST_2d_AIR","8"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","7"  "cust1","MM1","SHIP_CUST_2d_AIR","9"    vmi_to_shipcust flow    null    5   null    null    null
"vmi1","MM1","VMI","8"  "cust1","MM1","SHIP_CUST_2d_AIR","10"   vmi_to_shipcust flow    null    5   null    null    null
"cust1","MM1","SHIP_CUST_2d_AIR","2"    "cust1","MM1","CUST","2"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_2d_AIR","3"    "cust1","MM1","CUST","3"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_2d_AIR","4"    "cust1","MM1","CUST","4"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_2d_AIR","5"    "cust1","MM1","CUST","5"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_2d_AIR","6"    "cust1","MM1","CUST","6"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_2d_AIR","7"    "cust1","MM1","CUST","7"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_2d_AIR","8"    "cust1","MM1","CUST","8"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_2d_AIR","9"    "cust1","MM1","CUST","9"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM1","SHIP_CUST_2d_AIR","10"   "cust1","MM1","CUST","10"   shipcust_to_cust    flow    null    0   null    null    null
"source","-","-","0"    "vmi2","MM2","VMI","2"  source_to_VMI   supply  equality supply 0   50  null    null
"vmi2","MM2","VMI","0"  "cust1","MM2","SHIP_CUST_1d_AIR","1"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","1"  "cust1","MM2","SHIP_CUST_1d_AIR","2"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","2"  "cust1","MM2","SHIP_CUST_1d_AIR","3"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","3"  "cust1","MM2","SHIP_CUST_1d_AIR","4"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","4"  "cust1","MM2","SHIP_CUST_1d_AIR","5"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","5"  "cust1","MM2","SHIP_CUST_1d_AIR","6"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","6"  "cust1","MM2","SHIP_CUST_1d_AIR","7"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","7"  "cust1","MM2","SHIP_CUST_1d_AIR","8"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","8"  "cust1","MM2","SHIP_CUST_1d_AIR","9"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","9"  "cust1","MM2","SHIP_CUST_1d_AIR","10"   vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","0"  "vmi2","MM2","VMI","1"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi2","MM2","VMI","1"  "vmi2","MM2","VMI","2"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi2","MM2","VMI","2"  "vmi2","MM2","VMI","3"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi2","MM2","VMI","3"  "vmi2","MM2","VMI","4"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi2","MM2","VMI","4"  "vmi2","MM2","VMI","5"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi2","MM2","VMI","5"  "vmi2","MM2","VMI","6"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi2","MM2","VMI","6"  "vmi2","MM2","VMI","7"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi2","MM2","VMI","7"  "vmi2","MM2","VMI","8"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi2","MM2","VMI","8"  "vmi2","MM2","VMI","9"  vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi2","MM2","VMI","9"  "vmi2","MM2","VMI","10" vmi_to_vmi_inv  flow    null    0   null    null    null
"vmi2","MM2","VMI","10" "SINK","-","-","100"    vmi_to_sink esacpe  null    100 null    null    null
"cust1","MM2","SHIP_CUST_1d_AIR","1"    "cust1","MM2","CUST","1"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_1d_AIR","2"    "cust1","MM2","CUST","2"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_1d_AIR","3"    "cust1","MM2","CUST","3"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_1d_AIR","4"    "cust1","MM2","CUST","4"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_1d_AIR","5"    "cust1","MM2","CUST","5"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_1d_AIR","6"    "cust1","MM2","CUST","6"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_1d_AIR","7"    "cust1","MM2","CUST","7"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_1d_AIR","8"    "cust1","MM2","CUST","8"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_1d_AIR","9"    "cust1","MM2","CUST","9"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_1d_AIR","10"   "cust1","MM2","CUST","10"   shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","CUST","9"    "SINK","-","-","100"    cust_to_sink    flow    demand  0   null    null    10
"vmi2","MM2","VMI","0"  "cust1","MM2","SHIP_CUST_2d_AIR","2"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","1"  "cust1","MM2","SHIP_CUST_2d_AIR","3"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","2"  "cust1","MM2","SHIP_CUST_2d_AIR","4"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","3"  "cust1","MM2","SHIP_CUST_2d_AIR","5"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","4"  "cust1","MM2","SHIP_CUST_2d_AIR","6"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","5"  "cust1","MM2","SHIP_CUST_2d_AIR","7"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","6"  "cust1","MM2","SHIP_CUST_2d_AIR","8"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","7"  "cust1","MM2","SHIP_CUST_2d_AIR","9"    vmi_to_shipcust flow    null    1.4 null    null    null
"vmi2","MM2","VMI","8"  "cust1","MM2","SHIP_CUST_2d_AIR","10"   vmi_to_shipcust flow    null    1.4 null    null    null
"cust1","MM2","SHIP_CUST_2d_AIR","2"    "cust1","MM2","CUST","2"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_2d_AIR","3"    "cust1","MM2","CUST","3"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_2d_AIR","4"    "cust1","MM2","CUST","4"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_2d_AIR","5"    "cust1","MM2","CUST","5"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_2d_AIR","6"    "cust1","MM2","CUST","6"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_2d_AIR","7"    "cust1","MM2","CUST","7"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_2d_AIR","8"    "cust1","MM2","CUST","8"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_2d_AIR","9"    "cust1","MM2","CUST","9"    shipcust_to_cust    flow    null    0   null    null    null
"cust1","MM2","SHIP_CUST_2d_AIR","10"   "cust1","MM2","CUST","10"   shipcust_to_cust    flow    null    0   null    null    null
"vmi1","MM1","VMI","0"  "cust3","MM1","SHIP_CUST_1d_AIR","1"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","1"  "cust3","MM1","SHIP_CUST_1d_AIR","2"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","2"  "cust3","MM1","SHIP_CUST_1d_AIR","3"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","3"  "cust3","MM1","SHIP_CUST_1d_AIR","4"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","4"  "cust3","MM1","SHIP_CUST_1d_AIR","5"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","5"  "cust3","MM1","SHIP_CUST_1d_AIR","6"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","6"  "cust3","MM1","SHIP_CUST_1d_AIR","7"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","7"  "cust3","MM1","SHIP_CUST_1d_AIR","8"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","8"  "cust3","MM1","SHIP_CUST_1d_AIR","9"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","9"  "cust3","MM1","SHIP_CUST_1d_AIR","10"   vmi_to_shipcust flow    null    15  null    null    null
"cust3","MM1","SHIP_CUST_1d_AIR","1"    "cust3","MM1","CUST","1"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_1d_AIR","2"    "cust3","MM1","CUST","2"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_1d_AIR","3"    "cust3","MM1","CUST","3"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_1d_AIR","4"    "cust3","MM1","CUST","4"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_1d_AIR","5"    "cust3","MM1","CUST","5"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_1d_AIR","6"    "cust3","MM1","CUST","6"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_1d_AIR","7"    "cust3","MM1","CUST","7"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_1d_AIR","8"    "cust3","MM1","CUST","8"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_1d_AIR","9"    "cust3","MM1","CUST","9"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_1d_AIR","10"   "cust3","MM1","CUST","10"   shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","CUST","6"    "SINK","-","-","100"    cust_to_sink    flow    demand  0   null    null    5
"vmi1","MM1","VMI","0"  "cust3","MM1","SHIP_CUST_2d_AIR","2"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","1"  "cust3","MM1","SHIP_CUST_2d_AIR","3"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","2"  "cust3","MM1","SHIP_CUST_2d_AIR","4"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","3"  "cust3","MM1","SHIP_CUST_2d_AIR","5"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","4"  "cust3","MM1","SHIP_CUST_2d_AIR","6"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","5"  "cust3","MM1","SHIP_CUST_2d_AIR","7"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","6"  "cust3","MM1","SHIP_CUST_2d_AIR","8"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","7"  "cust3","MM1","SHIP_CUST_2d_AIR","9"    vmi_to_shipcust flow    null    15  null    null    null
"vmi1","MM1","VMI","8"  "cust3","MM1","SHIP_CUST_2d_AIR","10"   vmi_to_shipcust flow    null    15  null    null    null
"cust3","MM1","SHIP_CUST_2d_AIR","2"    "cust3","MM1","CUST","2"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_2d_AIR","3"    "cust3","MM1","CUST","3"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_2d_AIR","4"    "cust3","MM1","CUST","4"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_2d_AIR","5"    "cust3","MM1","CUST","5"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_2d_AIR","6"    "cust3","MM1","CUST","6"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_2d_AIR","7"    "cust3","MM1","CUST","7"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_2d_AIR","8"    "cust3","MM1","CUST","8"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_2d_AIR","9"    "cust3","MM1","CUST","9"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM1","SHIP_CUST_2d_AIR","10"   "cust3","MM1","CUST","10"   shipcust_to_cust    flow    null    0   null    null    null
"vmi2","MM2","VMI","0"  "cust3","MM2","SHIP_CUST_1d_AIR","1"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","1"  "cust3","MM2","SHIP_CUST_1d_AIR","2"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","2"  "cust3","MM2","SHIP_CUST_1d_AIR","3"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","3"  "cust3","MM2","SHIP_CUST_1d_AIR","4"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","4"  "cust3","MM2","SHIP_CUST_1d_AIR","5"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","5"  "cust3","MM2","SHIP_CUST_1d_AIR","6"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","6"  "cust3","MM2","SHIP_CUST_1d_AIR","7"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","7"  "cust3","MM2","SHIP_CUST_1d_AIR","8"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","8"  "cust3","MM2","SHIP_CUST_1d_AIR","9"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","9"  "cust3","MM2","SHIP_CUST_1d_AIR","10"   vmi_to_shipcust flow    null    1.8 null    null    null
"cust3","MM2","SHIP_CUST_1d_AIR","1"    "cust3","MM2","CUST","1"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_1d_AIR","2"    "cust3","MM2","CUST","2"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_1d_AIR","3"    "cust3","MM2","CUST","3"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_1d_AIR","4"    "cust3","MM2","CUST","4"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_1d_AIR","5"    "cust3","MM2","CUST","5"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_1d_AIR","6"    "cust3","MM2","CUST","6"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_1d_AIR","7"    "cust3","MM2","CUST","7"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_1d_AIR","8"    "cust3","MM2","CUST","8"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_1d_AIR","9"    "cust3","MM2","CUST","9"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_1d_AIR","10"   "cust3","MM2","CUST","10"   shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","CUST","8"    "SINK","-","-","100"    cust_to_sink    flow    demand  0   null    null    7
"vmi2","MM2","VMI","0"  "cust3","MM2","SHIP_CUST_2d_AIR","2"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","1"  "cust3","MM2","SHIP_CUST_2d_AIR","3"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","2"  "cust3","MM2","SHIP_CUST_2d_AIR","4"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","3"  "cust3","MM2","SHIP_CUST_2d_AIR","5"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","4"  "cust3","MM2","SHIP_CUST_2d_AIR","6"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","5"  "cust3","MM2","SHIP_CUST_2d_AIR","7"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","6"  "cust3","MM2","SHIP_CUST_2d_AIR","8"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","7"  "cust3","MM2","SHIP_CUST_2d_AIR","9"    vmi_to_shipcust flow    null    1.8 null    null    null
"vmi2","MM2","VMI","8"  "cust3","MM2","SHIP_CUST_2d_AIR","10"   vmi_to_shipcust flow    null    1.8 null    null    null
"cust3","MM2","SHIP_CUST_2d_AIR","2"    "cust3","MM2","CUST","2"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_2d_AIR","3"    "cust3","MM2","CUST","3"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_2d_AIR","4"    "cust3","MM2","CUST","4"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_2d_AIR","5"    "cust3","MM2","CUST","5"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_2d_AIR","6"    "cust3","MM2","CUST","6"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_2d_AIR","7"    "cust3","MM2","CUST","7"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_2d_AIR","8"    "cust3","MM2","CUST","8"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_2d_AIR","9"    "cust3","MM2","CUST","9"    shipcust_to_cust    flow    null    0   null    null    null
"cust3","MM2","SHIP_CUST_2d_AIR","10"   "cust3","MM2","CUST","10"   shipcust_to_cust    flow    null    0   null    null    null
"SINK","-","-","100"    "source","-","-","0"    closed_loop flow    null    0   null    null    null

My R code is as below: You have to take above csv data and paste it in file C:/dumy_network.csv.

My R code might not be very efficient but it serves the purpose!

library("lpSolve", lib.loc="C:/Users/njog/Documents/R/win-library/3.0")

#get the data from CSV file
mydata <- read.csv("C:/dumy_network.csv", header=TRUE)

#build list of nodes (no repetition)
nodes=unique(c(as.character(mydata$startlink),as.character(mydata$endlink)))
#list of all links
links=mydata[,1:2]
#cost of moving material on each link - optimization problem objective coefficients
cost=mydata$cost

#decision variable is flow on each link. Objective is to minimize product of (cost on each link*flow on the link). Therefore, count of decision variable is equal to count of links.

#constraints matrix: for each node in nodes, incoming quantity should be equal to outgoing quantity. 

constraints=matrix(0,sum(mydata$max_const!='null')+sum(mydata$equality_const!='null')+length(nodes),length(mydata$endlink))

for (i in 1:length(nodes) ) {
  constraints[i,]=t(1*(nodes[i]==links[,1])-1*(nodes[i]==links[,2]))
}

#get constraints for equality constraints- in some cases we have to ship material exactly same as this quanity.
constraint1=matrix(mydata$equality_const,1,length(mydata$equality_const))
constraint1[constraint1=="null"]=0
constraint1=as.numeric(constraint1)
constraint1_length=which(constraint1!=0)

constraint1_final=matrix(0,length(constraint1_length),length(mydata$equality_const))

for (i in 1:length(constraint1_length) ) {
  constraint1_final[i,constraint1_length[i]]=1
}
start=length(nodes)+1
end=length(nodes)+length(constraint1_length)
constraints[start:end,]=constraint1_final

#get constraints for maxconstraints - in some cases we cannot ship material exceeding this quanity.
constraint2=matrix(mydata$max_const,1,length(mydata$max_const))
constraint2[constraint2=="null"]=0
constraint2=as.numeric(constraint2)
constraint2_length=which(constraint2!=0)

constraint2_final=matrix(0,length(constraint2_length),length(mydata$max_const))

for (i in 1:length(constraint2_length) ) {
  constraint2_final[i,constraint2_length[i]]=1
}
start=end+1
end=end+length(constraint2_length)
constraints[start:end,]=constraint2_final


#building direction of constraints
direction=c(rep("=",length(nodes)),rep("=",sum(mydata$equality_const!='null')),rep("<=",sum(mydata$max_const!='null')))

#building right hand side of constraints
b1=as.numeric(as.character(mydata$equality_const[constraint1_length]))
b2=as.numeric(as.character(mydata$max_const[constraint2_length]))
b=c(rep(0,length(nodes)),b1,b2)

res = lpSolve::lp('min', cost, constraints, direction, b,  all.int = TRUE)
res$solution

answers1=data.frame(res$solution)
mydata=cbind(mydata,answers1)

=====================================update 2====================================== Not getting any answers, so trying to simplify my question:

Examples section of the page gives a simple problem. Does anyone has ideas how to solve it using Mapreduce? I mean let's say I have a similar problem ,but with a huge number of variables and constraints then is there a way to achieve faster processing?

Usually distributed optimization involves some sort of a decomposition approach (e.g. Benders' decomposition or Dantzig-Wolfe decomposition). We'll need more details about your model to help you. — josliber, Apr 04 '14 at 17:50
simplified version of my problem is at http://stackoverflow.com/questions/22736107/network-directed-graph-optimization-package-in-r. My actual model has many different nodes and travel options between 2 cities... — user2543622, Apr 04 '14 at 18:06
Please edit your question to add a minimum reproducible example of your data and the code you're currently running. — josliber, Apr 04 '14 at 18:18
@josilber I have added info as per your request. I hope it makes sense. Looking forward to your input! — user2543622, Apr 04 '14 at 19:00
It looks like a parallel approach to the transportation problem has been proposed: http://www.dtic.mil/dtic/tr/fulltext/u2/a196238.pdf. Implementing such an approach will be quite involved, though. — josliber, Apr 05 '14 at 02:30
Hadoop is mostly used for map-reduce, which what you're looking for is distributed optimization. I've never heard of somebody implementing a distributed optimization approach using Hadoop, because it's really the wrong tool for the job. Perhaps your best bet is to use a commercial tool like Gurobi or Cplex, which have better LP solvers and can take advantage of multiple processors on the same computer. Both can be used for free with an academic license if you're at a university. — josliber, Apr 09 '14 at 23:37
I am using cplex (IBM ILOG SOFTWARE) currently. But it takes a lot of time to solve :( Therefore, i thought that hadoop might be a better approach — user2543622, Apr 10 '14 at 14:41
I think the cplex direction makes more sense -- I'm not sure lpsolve can even use multiple cores. I would suggest adding your cplex code and appropriate tags so it's visible to the cplex community on SO. Distributed optimization is hard -- there's a ton of inter-process communication that needs to take place. Again, hadoop is probably not the correct tool. — josliber, Apr 10 '14 at 14:55
in past i tried to get help on CPLEX..but didnt get good feedback...cplex community seems to be smaller than r community — user2543622, Apr 10 '14 at 18:55
Can you link the cplex question or expand the current question? As I said, I think `lpSolve` is a dead end for you. I have some expertise with cplex, so I might be able to help. — josliber, Apr 10 '14 at 19:15

Markus · Answer 1 · 2014-04-15T06:49:17.513

2

In short, you want to do linear programming (lp), in large scale and you are not happy with the performance of your solver.

Approaches

I suggest the following approaches.

Solver

Do you need to use this solver? Here are some alternatives:
- Gurobi
- Mosek
- CPlex

Parallel execution

A weak approach to achiev parallel processing is to run process in parallel. Can you launch several optimations at once? (Here, you could write a simple exectuion task in Hadoop)

Improved inputs

The solvers can perform much better if you hand in the data accordingly (sparse matrix, ...)

In your code you use lpSolve::lp. According to the CRAN page of lpSolver 5.6.7 and its PDF, there is a special mode for the transportation probelm lpSolve:lp.transport. (I haven't worked with R and I am not familiar with the syntax. I am a Matlab "enthusiast".)

Latest research

Maybe some engergy is spent in academia. You might find a paper (cost some dollars) on sciencedirect.com.

Hardware

The faster the CPU, the faster your get a soluation (answer). Do you mind giving us some insights on your environment (OS, CPU)?

Final thought

Have you tried to raise your question in quant.stackexchange.com because numerical optimization is closely related to mathematics and "does not involve that much programming".

Parallel mode

We use the CLP. According some colleagues you can set a flag or use a different implementation to run the solver in parallel (1 problem executed on many cores). Take a look at SYMPHONY.

Keep us posted and do you mind giving us some insights on your environment (OS, CPU)?

My bounty

On Quora you find your answer that Hadoop is good in data processing, fast in I/O, not specially designed for optimization or calculation: Quora

Please let me know, if you do not want to award me with the bounties.

Missing bounty of +100

I have not received any bounties even though I did some research... This thread comes to a point where you do not give us the necessary details and we should solve it for you - this never gonna work out!

edited Apr 15 '14 at 06:49

answered Apr 13 '14 at 13:05

Markus

763
7
24

solver: would like to stick to lpsolve solver only. But i dont think that changing a solver going to have a huge impact, parallel execution: not possible - entire network has to be run on a single pc - one node can supply to more than 2 stations and 1 station can receive from more than one node, improved inputs: will give it a shot - but dont think it will have huge impact, latest research: will try to give it a shot, hardware: i am using the fastest config available to me, final thought: will try – user2543622 Apr 13 '14 at 17:49
Lately, we did a ALM optimization for a large, top-level inscurance and we spent several days figuring out what's the best solver to speed up the calculations. For a different project we bought a new powerful machine to check how a solver scales in parallel mode and a certain OS... CPU, Solver and OS have a closer relationship than (we) you think. – Markus Apr 13 '14 at 17:55
you have mentioned solver scales in parallel mode ...what do you mean by parallel mode? would it be possible to give some insight? – user2543622 Apr 13 '14 at 17:58
you have given a lot of information...let me read it and get back to you on bounty thing – user2543622 Apr 13 '14 at 20:30
is there a way i can assign you half bounty? i appreciate your inputs and would like to reward you. at the same time, as i havent got solution, i dont think that i should give full bounty – user2543622 Apr 14 '14 at 02:51
I think this must be addressed by the admin... Have you tried to run it on SYMPHONY? Do you mind give us further details (OS, CPU, Client Server application,...)? – Markus Apr 14 '14 at 18:42
I read wikipedia SYMPHONY article and also looked at the website..ti seems that preparing my data to run on symphony will take a lot of time..i am new to it and sadly dont have too much of time to explore SYMPHONY...apart from that i have a 50 GB ram available and windows OS... – user2543622 Apr 14 '14 at 23:14
Nativ or virtualized? Do you reach the RAM barrier in your machine? (As repeated many times: give us some details about your environment!) – Markus Apr 15 '14 at 06:47
it is on an office server ....so i guess that means virtualized. During ILOG program we reach RAM barrier. By environment what are you looking for? IMHO contradictory to my earlier thinking, the solution is very complicated and will need a lot of research...i might not be able to dedicate a lot of time to the problem...appreicate your help.. – user2543622 Apr 15 '14 at 14:25