How to design highly scalable web services in Java?

Question

I am creating some Web Services that would have 2000 concurrent users. The services are offered for free and are hence expected to get a large user base. In the future it may be required to scale up to 50,000 users.

There are already a few other questions that address the issue like - https://stackoverflow.com/questions/2567254/building-highly-scalable-web-services

However my requirements differ from the question above.

For example - My application does not have a user interface, so images, CSS, javascript are not an issue. It is in Java so suggestions like using HipHop to translate PHP to native code are useless.

Hence I decided to ask my question separately.

This is my project setup -

1. Rest based Web services using Apache CXF
2. Hibernate 3.0 (With relevant optimizations like lazy loading and custom HQL for tune up)
3. Tomcat 6.0
4. MySql 5.5

What are the best practices to abide by in order to make a Java based application scalable?

Answer 1

I dealt with the issue in the past, but still feel I have lots to learn on the field. I find this to be one of the most interesting fields there are in software development nowadays, here are some thoughts about this:
MySQL is fair enough database unless you're working with massively huge amount of data, and in this case you might consider NoSQL database, but you should carefully examine what is the best NoSQL database for your needs.

You should implement caching at your system - try to cache as much read-only data as much as possible, or define some caching strategies - for example, we had a scenario in which it was valid for a user to see "old data" as long as the recent update took place in the last hour.
I would consider JBoss Cache, or maybe Infinispan (which is more like a distributed data structure) or other popular caching framework for this.
In addition, as you mentioned tomcat, I assume you work in some request-respone module. Try to consider using a cache that exists in a scope of a given request, this can be even a simple HashMap that is associated with the thread local storage.
My idea here quite resembles to first level cache at Hibernate.

You should remember that files, transactions and other resources are expensive in terms of keeping them open. Make sure you close files and transactions as soon as possible, or you will end up with bugs that will reproduce on large scale setups

In addition you must understand what 2000 concurrent users - does this mean that 2000 users are accessing your server at once, or are they using your system? Distinguish between cases where 2000 users try to open a socket to your server, and a case where only 500 are, and 1500 are currently looking at results, of filling input at client side.

You should consider using clustering - you will have to deal with issues like load balancing, sticky session (which means the load balancer will redirect a request to the same server for the same session) and more.

If you need to have synchronization code - choose synchronization strategy carefully. I saw some systems in which a simple lock was used, but a ReaderWriterLock could have improved things, as most access was read-only.

Consider having client side caching and validation if possible, try to save calls to server, and to send only differences of data, in case most of your response for a request with same parameter does not change.
For example, at oVirt open source project we request to get statistics of a given virtual machine. some of the data of the VM rarely changes, so we send only MD5 of it, if the data changes the MD5 value is changed as well , we perform a request to get the full data, and not just the MD5.

I mentioned hibernate before - I would reocmmend you to carefully consider using it - if you need to perform lots of writes, and less reads, Hibernate might not be ideal for you, and you should consider maybe working with Spring-JDBC as a wrapper over JDBC.

Index your database wisely, and use a correct db scheme. Consider using a layer of stored procedures as they are precompiled and optimized

I would like to state that at past, I dealt with a system (single node) on mysql (mostly read only access) with jboss 4.2.1 and managed to reach 2000 concurrent users
(not accessing at once in terms of opening 2000 sockets against our server), but using/browsing our system, using JBoss Cache and preloading to the cache some of the most accessed data, or data we realized is going to be "hot and popular" but our solution was good for our architecture and our flows,
so as I say in these cases -
There are more tips and tricks , but it really depends on your architecture, and what flows you need to have in your system. Good luck!

Answer 2

Good question. Probably hard to say which is best approach, but will try from my experience.

The best way to scale the Java based web application is to write it as stateless as possible (if you could). This allows you to horizontally scale the application, where you can add tomcat servers if there are more concurrent users.

However, as you noted, there could be issue with the database connections. But the question I have is, how are you getting the data? Is it user generated or you get the data from third party? This is very important because, if you are giving a service to your user with the data aggregated from third party application (say FB, Twitter etc), then what you can follow, is write to master database and replicate the data over to slave databases which are allocated to each tomcat instances. Then each tomcat server can get from its own slave database.

 Are there faster alternatives to Mysql?

You can go for MySQL cluster which have in-memory datastore. But beware of the fact that the application may need some changes. The sql joins are not well supported in MySQL cluster though in the latest version there are improvements for the same. If the cost is not a factor, then you can try Oracle.

The caching solution definitely will improve the performance. But then, it all depends on the architecture of the whole application. You should be well aware of when to push data to the cache, when to make it dirty(remove from cache).

Regarding distributing the load in multi server environment, I would suggest you to use load balancer than to use Apache for load balancing.

Answer 3

I'm currently setting up a similar system (on a professional level) and this is the design I've chosen:

• Two Nginx loadbalancers (both active, both failover for the other, balanced with DNS round robin)
• Two MySQL Databases in master master replication mode
• Two Tomcat instances as a tomcat cluster
• Two Memcached instances for both caching and session state sharing for the Tomcat cluster

This will achieve a redundant, high availability, scalable solution.

The loadbalancers (on decent hardware) will easily loadbalance a saturated 1gbit line each. This is also a great place for SSL offloading.

You can save your session info in memcached. In case a tomcat instance fails, another tomcat instance can retrieve relevant session information and the clients won't notice a thing. Don't forget to combine this with sticky sessions too. (To keep network traffic down)

Tomcat clustering also has an option to share session information among the cluster in real time, without using memcached. Although I think performance wise, using Memcached will be better.

If you need more power in any of these applications:

• Nginx: Add more loadbalancers, although I don't think this will be the bottleneck very soon.
• Tomcat: you can easily increase the size of the Tomcat cluster or add more clusters
• Mysql: Add some read-only slaves or increase the cluster size (depending on your application, but since you wrote a REST-based application, this should not be a problem)
• Memcached: Add more nodes, Memcached scales pretty well I believe.

I don't know how your application is build and what the big resources hogs are, but if you see a high database load (during your loadtests!), adding a cache between the application and database could certainly improve performance a lot. But don't forget that not everything is cachable, if your queries are always different, caching won't help (much)

My advice would be to download VMware Workbench (or similair virtualization software) and try to create a simple setup. No loadbalancing or clustering, just the basics and work from there. One by one add more features (balancing, caching, clustering, etc.) and make sure to do some research on each topic, so you'll know you made the right pick.

If you keep running the same performance tests during this process, you can see for yourself if using X is better than using Y in your setup, or what impact caching will have, etc.

In the end, a setup like this really depends on the requirements of your application and its clients, everything can be done in various ways, each with it's own strengths and weaknesses.

Any more questions?

Good luck!

Wesley