Stack Overflow
Stack Overflow

Questions tagged [large-scale]

34 questions
562
votes
8 answers

Large-scale design in Haskell?

What is a good way to design/structure large functional programs, especially in Haskell? I've been through a bunch of the tutorials (Write Yourself a Scheme being my favorite, with Real World Haskell a close second) - but most of the programs are…
Dan
  • 5,763
  • 3
  • 17
  • 13
13
votes
2 answers

Applying a function in each row of a big PySpark dataframe?

I have a big dataframe (~30M rows). I have a function f. The business of f is to run through each row, check some logics and feed the outputs into a dictionary. The function needs to be performed row by row. I tried: dic = dict() for row in…
mommomonthewind
  • 4,390
  • 11
  • 46
  • 74
10
votes
3 answers

Dealing with large files in Haskell

I have a large file (4+ gigs) of, lets just say, 4 byte floats. I would like to treat it as List, in the sense that I would like to be able to use map, filter, foldl, etc. However, instead of producing a new list with the output, I would like to…
Jonathan Fischoff
  • 1,467
  • 12
  • 22
8
votes
3 answers

How to solve large-scale nonlinear optimization problems with Ceres?

I need to optimize a surface represented by a 2D grid of points to produce normal vectors of the surface that align with provided target normal vectors. The grid size is likely to be between 201x201 and 1001x1001. That means that the number of…
Eric
  • 244
  • 5
  • 10
7
votes
8 answers

Brain modelling

Just wondering, since we've reached 1 teraflop per PC, yet we are still not able to model an insect's brain. Has anyone seen a decent implementation of a self-learning, self-developing neural network?
Andy
  • 2,670
  • 3
  • 30
  • 49
4
votes
1 answer

Whats the best structure for a large scale python tornado project?

I'm using a mongodb database backend with tornado as my core. I currently just have my main file with a bunch of handlers in it. It's a multiuser webapp with links between users, aka a "friend" system. My current structure is: templates/ …
carboncomputed
  • 1,630
  • 3
  • 20
  • 42
3
votes
0 answers

How to fan-out/distribute 500,000 records with Firebase?

Requirements: Users can register to company & unsubscribe freely. Users should get only the posts of companies they are registered to. Users shouldn't see posts they already liked/disliked Currently, what we are doing is, when a user joins a…
Uri Brodsky
  • 59
  • 4
3
votes
1 answer

Optimizing a query for optional fields from another table

I have a innodb table called items that powers one ecommerce site. The search system allows you to search for optional/additional fields, so that you can e.g. search for only repaired computers or cars only older than 2000. This is done via…
aez1
  • 31
  • 3
2
votes
3 answers

Is there any must-have kind of book about Symfony-2.0 and large scale projects?

I'm looking for a guide book for Symfony framework, specially version-2.0. I'm interested in more advanced guides, specially about how to "squeeze" the maximum performance from symfony, optimizations for medium to large scale projects where even…
marcio
  • 10,002
  • 11
  • 54
  • 83
2
votes
1 answer

Large-scale models and FMU for model exchange

I want to export large-scale models from Openmodelica to FMU for model exchange. As first step, I'm using this model for testing purposes: model BigModel "Tank-Pipe-Tank cloned n times" // constant Real g=9.8; // parameter Integer n = 1; …
Neotake
  • 79
  • 7
2
votes
0 answers

How to independently scale M web app servers and N cache servers?

I have M web app servers (with the same app on them, shared-nothing), behind a load balancer. The app running requires heavy caching, so I would like to have N cache servers behind (using Memcached). I would like to use consistent hashing, so I…
v_i_m
  • 171
  • 1
  • 2
  • 6
2
votes
4 answers

How to avoid large matrix multiplication in Matlab

There are two large matrices in my code, which have the same number of columns and different number of rows. Like A(20000X4000) and B(30000X4000). Both are 0-1 sparse. I should check each row of A with all rows of B and count the number of common…
Alef
  • 41
  • 1
  • 7
2
votes
1 answer

Large shared code base, extend functionality

Our application is a large scale Javascript application, using the Javascript MVC framework. We are using the MVC app folder across all of our sites via an SVN:external, and each site also has their own files. The settings files are specific to the…
Scott Warren
  • 243
  • 3
  • 12
2
votes
2 answers

Memory problems with large-scale social network visualization using R and Cytoscape

I'm relatively new to R and am trying to solve the following problem: I work on a Windows 7 Enterprise platform with the 32bit version of R and have about 3GB of RAM on my machine. I have large-scale social network data (c. 7,000 vertices and c.…
user1326028
  • 81
  • 7
1
vote
2 answers

Scalable System Architecture/Design for Reading/Parsing Files

Background: I am designing a software application that reads millions or much more files and either converts or just parses those files. Part of requirement is to build a scalable and distributed system so that reading and parsing can be scaled…
santosh
  • 13
  • 3
1
2 3