5 seconds is relatively slow for a SQL Query, indicating that its doing a fair amount of work (e.g. a lot of disk IO). A query involving a lot of disk IO is likely to run slower when executed by several concurrent users because of increased load on the SQL Server. You should get an execution plan and look at whether the 5s query can be optimized, e.g. with additional indexes.
Its also possible that locking is to blame, e.g. if you query exclusive locks the entire table and takes 5s to run, a second query run concurrently would need to wait 5s for the original query to finish executing, and would then lock the table itself and take an additional 5s to execute resulting in a 10s execution time. 3 concurrent queries would take 15s etc... See How to identify blocking in SQL Server for a bunch of methods to check whether or not this is the case.