2

I have C applications that will run on multiple machines at different sites.

Now I want to control and monitor these C applications. For that I am thinking about Java Web Application using Servlet/JSP.

I am thinking that C applications will connect to Java Web application over TCP. In my web application, I am thinking to implement manager which communicates with C applications over TCP. I will start manager when web application starts as separate thread. And manager will communicate to servlet requests via Context and Session. So whenever user do something on browser, I want to use functionalities of my manager at server, with ServetContext an Session as interface.

So this is what I am thinking. So, I want to know if there is better approach, or I am doing anything wrong? Can anyone please suggest me better solution?

EDIT

Current workflow: whenever I need to start / stop C application, I have to SSH remote machine puTTY terminal, type long commands, and start / stop it. Whenever there is some issue, I have to scroll long long log files. There couple of other things like live status of what application is doing/processing all things at every second, that I can't log always in log file.

So I find these workflow difficult. And things like live status I can't monitor.

Now I want to have web application interface to it. I can modify my C application and implement web application from scratch.

New Workflow to implement: I want to start / stop C application from web page. I want to view logs and live status reports / live graphs on web page (monitoring what C application is doing). I want to monitor machine status also on web page.

The web interface I thinking to design in Java using JSP/servlets.

So, I will modify my C application so it can communicate with with web application.

Question:

Just need guidelines / best practices for making new workflow.

EDIT 2

Sorry for confusion between controller or manager. Both are same thing.

My thoughts: System will consist of C applications running at different sites, Java controller and Java web app running parallely in Tomcat server, and DB.

1) C applications will connect to controller over TCP. So, controller here becomes server and C applications client.

2) C applications will be multithreaded, will receive tasks from controller and spawns new thread to perform that task. When controller tells to stop task, C application will stop thread of that task. Additionally, C applications will send work progress (logs) every second to controller.

3) Controller receives task commands from web application (as both running parallelly in Tomcat server, both in same instance on JVM), and web application will receive commands from user over HTTP.

4) The work progress (logs) received every second from C applications to controller, controller will then insert logs in DB for later analysis (need to consider if it is good insert logs in MySQL RDBMS, may be needed to do lot of inserts, may be 100 or 1000 every second, forever). Web application may also request recent 5 minute logs from controller and send to user over HTTP. If user is monitoring logs, then web application will have to retrieve logs every second from controller and send to user over HTTP.

5) User monitoring C application tasks, will see progress in graph, updated every second. Additionally text lines of logs of info/error events that may happen occasionally in C applications.

6) C applications will be per machine, which will execute any task user sends from web browser. C applications will be running as service in machine, which will start on machine startup, will connect to server, and will stay connected to server forever. Can be running idle if no tasks to perform.

brb tea
  • 345
  • 1
  • 12
  • Let me know if my question is not clear. – brb tea Nov 14 '14 at 17:28
  • So, basically, there will be a different thread in your Java web app that will collect the data from the C apps and then you web app will query this data from the thread. Right? – Luiggi Mendoza Nov 14 '14 at 17:55
  • @LuiggiMendoza yes, I am thinking to implement manager (controller) that communicates with web application with ServletContext/HttpSession and communicates with C applications over TCP connections. And I will start manager in separate thread when web application starts. May be starting it in contextInitialized event. – brb tea Nov 14 '14 at 18:01
  • Just a tip: don't use ServletContext nor Session to store this data. Instead, use a third component like a database (relational or in-memory, probably in-memory would fit better for faster access to the data) since none of them is a real cache and they won't support multi threading addition/removal of elements. Also, I would split the manager app (that receives data from C apps) and the web app. – Luiggi Mendoza Nov 14 '14 at 19:10
  • @LuiggiMendoza sure, I will have DB to store data. But to share some recent log data, I am thinking to share directly to web app (synchronized). – brb tea Nov 14 '14 at 20:31
  • @LuiggiMendoza I have updated my question for more details. The controller (manager) and web app will be running in same tomcat server. The controller will be executing in its own in separate thread, which will be started/stopped in context initialized and destroyed events, and which may spawn other threads. – brb tea Nov 14 '14 at 20:32
  • @LuiggiMendoza The servlet requests will retrieve recent logs data from controller in synchronized manner, as both will be separate threads. – brb tea Nov 14 '14 at 20:38
  • You might be looking for named pipes (http://v01ver-howto.blogspot.fr/2010/04/howto-use-named-pipes-to-communicate.html) also known as FIFO (http://man7.org/linux/man-pages/man7/fifo.7.html) – Lectem Nov 15 '14 at 21:09

2 Answers2

2

It is a valid approach, I believe sockets is how most distributed systems communicate, and more often than not even different services on the same box communicate that way. Also I believe what you are suggesting for the java web service is very typical and will work well (It will probably grow in complexity beyond what you are currently thinking, but the archetecture you describe is a good start).

If your C services are made to also run independantly of the management system then you might want to reverse it and have the management system connect to the services (Unless your firewall prevents it).

You will certainly want a small, well-defined protocol. If you are sending lots of fields you could even make everything you send JSON or xml since they will already have parsers to validate the format.

Be careful about security! On the C side ensure that you won't get any buffer overflows and if you parse the information yourself, be strict about throwing away (and logging!) data that doesn't look right. On Java the buffer overruns aren't as much of a problem but be sure that you log packets that don't fit your protocol exactly to detect both bugs and intrusions.

Another solution that you might consider--Your systems all share a database already you could send commands and responses through the DB (Assuming the command/responses are not happening too often). We don't do this exactly, but we share a variable table in which we place name/value pairs indicating different aspects of our systems performance and configuration (it's 2-way), this is probably not optimal but has been amazingly flexible since it allows us to reconfigure our system at runtime (the values are cached locally in each service and re-read/updated every 30 seconds).

I might be able to give you more info if I knew more specifics about what you expected to do--for instance, how often will your browser update it's fields, what kind of command signals or data requests will be sent and what kind of data do you expect back? Although you certainly don't have to post that stuff here, you must consider it--I suggest mocking up your browser page to start.

edits based on comments: Sounds good, just a couple comments:

2) Any good database should be able to handle that volume of data for logging but you may want to use a good cache on top of your DB.

5) You will probably want a web framework to render the graph and manage updates. There are a lot and most can do what you are saying pretty easily, but trying to do it all yourself without a framework of some sort might be tough. I only say this because you didn't mention it.

6) Be sure you can handle dropped connections and reconnecting. When you are testing, pull the plug on your server (at least the network cable) and leave it out for 10 minutes, then make sure when you plug it back in you get the results you expect (Should the client automatically reconnect? Should it hold onto the logs or throw them away? How long will it hold onto logs?)

You may want to build in a way to "Reboot" your C services. Since they were started as a service, simply sending a command that tells them to terminate/exit will generally work since the system will restart them. You may also want a little monitoring loop that restarts them under certain criteria (like they haven't gotten a command from the server for n minutes). This can come in handy when you're in california at 10am trying to work with a C service in Austraillia at 2am.

Also, consider that an attacker can insert himself between your client and server. If you are using an SSL socket you should be okay, but if it's a raw socket you must be VERY careful.

Correction:

You may have problems putting that many records into a MySQL database. If it is not indexed and you minimize queries against it you may be okay. You can achieve this by keeping the last 5 minutes of all your logs in memory so you don't have to index your database and by grouping inserts or having a very well tuned cache.

A better approach might be to forgo the database and just use flat log files pre-filtered to what a single user might want to see, so if the user asks for the last 5 minutes "WARN" and "DEBUG" messages from a machine you could just read the logfile from that machine into memory, skipping all but warn/debug messages, and display those. This has it's own problems but should be more scalable than an indexed database. This would also allow you to zip up older data (that a user won't want to query against any more) for a 70-90% savings in disk space.

Bill K
  • 62,186
  • 18
  • 105
  • 157
  • Thanks a lot for valuable answer. Yes so I am starting roll on TCP connections between controller and C applications. The DB approach also seems good. But also C applications will be at different sites, like in Australia, Canada, USA, India, so will DB be helpful here? – brb tea Nov 14 '14 at 20:22
  • Absolutely I have to consider protocol between C applications and controller, that will be fail-proof against version increments. – brb tea Nov 14 '14 at 20:25
  • Can you please look at my updated question, and please provide some suggestions? – brb tea Nov 14 '14 at 20:44
  • Thanks again for useful tips. The logs generated every second, will be some sort of samples at every second, like send/receive data rates, number of various kind of data errors, etc. Yes large number of DB inserts doesn't seem fine to me, it can be okay right now, but who knows future. I am thinking to compress logs of duration say 5 minutes or hour as BLOB or TEXT, and insert in DB indexed by datetime. – brb tea Nov 15 '14 at 06:51
  • Regarding (6), yes I will need mechanisms for reconnecting TCP connections, and restore state. I will need mechanisms in C applications to keep logs in temporary files while TCP connection is down. I will need to heartbeat C applications and Controller. – brb tea Nov 15 '14 at 06:55
  • Regarding security, I will make each C application to authenticate itself with username/password things, but connections will be unsecured TCP right now. But will need think to use SSL avoid man-in-middle attacks. – brb tea Nov 15 '14 at 06:59
  • To restart services, or to update applications, I am thinking is: there will be wrapper C process, which will start my C application. Then I can send commands to reboot/exit to C application, and C application will terminates with status that wrapper process can know and will restart C application, or download new version, and replace C application image. And wrapper application will manage better in C application crashes if happens due to bugs. – brb tea Nov 15 '14 at 07:07
2

Here are my recommendations on your current design and since you haven't defined a specific scope for this project:

  • Define a protocol to communicate between your C apps and your monitor app. Probably you don't need the same info from all the C apps in the same format or there are more important metrics for some C apps than others. I would recommend using plain JSON for this and to define a minimum schema to fulfill in order for both C to produce the data and Java for consume and validate it.
  • Use a database to store the results of monitoring your C apps. The generic option would be using a RDBMS, probably open source like MySQL or PostgreSQL, or if you (or your company) can get the licenses go for SQL Server or Oracle or another one. This in case you need to maintain a history of the results, and you can clear the data periodically.
  • Probably you want/need to have the latest results from monitoring available in a sort of cache (because in this time performance is critical), so you may use an in-memory database like Hazelcast or Redis, or just a simple cache like EhCache or Infinispan. Storing the data in an external element is better than storing it in plain ServletContext because these technologies are aware of multi threading and support ACID, which is not the primary use case for ServletContext but seems necessary for the monitor.
  • Separate the monitor that will receive the data from the C apps from the web app. In case the monitor fails or it takes too much time to perform some operations, the Web application will still be available to work without having the overhead to receive and manage the data from the C apps. In the other hand, if the web app starts to be slower (due to problems in the implementation of the app or something that should be discovered using a profiler) then you may restart it, and by doing this your monitor should continue gathering the data from the C apps and store them in your data source.
  • For the threads in the monitor app, since it seems it will be based on Java, use ExecutorService rather than creating and managing the threads manually.

For this part:

User monitoring C application tasks, will see progress in graph, updated every second. Additionally text lines of logs of info/error events that may happen occasionally in C applications

You may use Rx Java to not update your view (JSP, Facelet, plain HTML or whatever you will use) or another reactive programming model like Play Framework to read the data continuously from database (and cache if you use it) and update the view in a direct way for the users of the web app. If you don't want to use this programming model, then at least use push technology like comet or WebSockets. If this part is not that important, then use a simple refresh timer as explained here: How to reload page every 5 second?

For this part:

C applications will be per machine, which will execute any task user sends from web browser

You could reuse the protocol to communicate the C apps using JSON to the monitor and another thread in each C app to translate the action and execute it.

Community
  • 1
  • 1
Luiggi Mendoza
  • 85,076
  • 16
  • 154
  • 332
  • Thanks a lot for your valuable suggestions. Yes I will have RDBMS and protocol between C apps and controller. The use in-memory databases seems better, instead having hashmaps/lists and managing it myself for thread-safeties. – brb tea Nov 15 '14 at 07:13
  • For *C applications will be per machine, which will execute any task user sends from web browser*, sorry I meant not directly from browser to C app communication, but via web application and controller. – brb tea Nov 15 '14 at 07:16
  • For *User monitoring C application tasks, will see progress in graph, updated every second. Additionally text lines of logs of info/error events that may happen occasionally in C applications*, right now I am thinking to use AJAX/jquery at client side, and web server pulling recent log data from in-memory DB. As most of time user will be monitoring live, I may not need MySQL accesses frequently. – brb tea Nov 15 '14 at 07:19
  • For separating web app and controller app, can you please suggest how will I make controller app communicate to web app? Web services? Web application may need to call controller lot of times, like 100 times/second. With separation you meant on separate JVM? – brb tea Nov 15 '14 at 07:23
  • @Huko the communication will be through database, both relational and in-memory. Since both support ACID transactions and support multi threadong there won't be problems for retrieving consistent data at any time you need it. – Luiggi Mendoza Nov 15 '14 at 08:46
  • But then controller will have to poll DB for actions/commands from web app continuously. Is that better? – brb tea Nov 15 '14 at 08:51
  • The controller will only receive the data to monitor and store ir in your data sources. The web app will query the relevant data from the data sources. – Luiggi Mendoza Nov 15 '14 at 08:55
  • Yes correct, in that scenario, controller will insert data directly to DB. But when user on browser do something (start/stop new task), it will go to web app, then controller and then to C apps. So I need way to invoke services of controller from web app. – brb tea Nov 15 '14 at 08:57
  • Please stop calling it controller. Controller seems more like C from MVC when this *controller* is more a monitor than a controller (because it doesn't have a real plain control, in fact is more like a middleware rather than a controller). Since the web app will only query the database, the monitor and your C apps won't be affected by the operations done in the web app, this approach adds complexity but gives you more flexibility. Also, if a C app or the monitor breaks, the web app can still work and consume the data from the database. – Luiggi Mendoza Nov 15 '14 at 09:02
  • Yes, sorry for my terminology, I can call it middleware. But the problem is, I will need some sort of web services so web app can invoke things in middleware asynchronously, instead of middleware polling DB to know what web app is asking. Is that correct? – brb tea Nov 15 '14 at 09:11
  • Thanks buddy for your valuable answer. I can't give bounty to more than one answer, but I will start bounty again to provide you. – brb tea Nov 16 '14 at 14:36