If your server strains a little more each month while traffic stays flat, your visitors may not be producing the load. These drains never appear in analytics reports; they show only in server graphs and on the invoice.
Short answer: what consumes the most server resources?
Beyond genuine visitor traffic, six sources consume the most: bot and crawler traffic, uncontrolled scheduled tasks, bloated log and cache files, unoptimised database queries, search and filter pages, and backup operations. What they share is invisibility in analytics — analytics counts only real users while the server serves everything. That gap is the basic explanation for "my traffic is the same but the server got slower".
Six hidden drains
| Source | Symptom | How to confirm |
|---|---|---|
| Bot and crawler traffic | High request counts, quiet analytics | User-agent distribution in access logs |
| Scheduled tasks | Sudden load at particular hours | Cron records against an hourly load graph |
| Log and cache bloat | Disk filling fast, writes slowing | Directory sizes listed largest first |
| Slow database queries | Long page generation, high CPU | Slow query log |
| Search and filter pages | Heavy, uncacheable requests | List of most-requested dynamic addresses |
| Backup operations | Congestion at night or set hours | Backup schedule compared with load graph |
The third surprises people most. Log files and cache folders grow for years because nobody looks at them; when the disk fills, the problem appears not as slowdown but as a database that suddenly stops working.
Separating bot traffic
- Wanted bots. Search engine crawlers and AI crawlers that use your content in answers. These are not blocked, though crawl rates can be tuned where needed.
- Neutral bots. Monitoring services, link checkers, screenshot services. They create load but cause no harm; rate-limit if necessary.
- Unwanted bots. Automated traffic scraping content, monitoring prices, scanning for flaws or attempting credentials. These should be blocked.
Do not rely on user-agent strings alone; unwanted bots often identify themselves as legitimate crawlers. Behaviour patterns — requesting the same page at very short intervals, ignoring robots.txt, generating very high volume from one IP — are more reliable. The natural place for that filtering is an application-layer firewall; we covered it in our WAF article.
Four quick-win interventions
- Configure caching correctly. The largest and fastest gain sits here. Caching pages generated dynamically but changing rarely reduces CPU load markedly. The one caution: price and stock data must never be served from cache.
- Rotate logs and temporary files. Set up automatic log rotation, clear old backups, put size limits on cache folders. This eliminates disk-driven outages entirely.
- Spread scheduled tasks. If every job runs at midnight they all run at once and strain the server together. Distributing them across different minutes produces measurable relief on its own.
- Find and index heavy queries. Leaving the slow query log on for a week usually reveals the two or three queries producing most of the load. The fix is often adding a missing index.
These four belong before any hardware upgrade. Before deciding on capacity, you need to know how much of the current load comes from real visitors — the method is in our capacity testing article.
Netişlem expert view: examine the load's source before upgrading the plan
When businesses come to us saying "our server is not enough, let's move up a plan", we analyse the load first. In a significant share of cases, a noticeable portion of it does not come from real visitors: scraping traffic, uncontrolled cron jobs, or a single unoptimised query. Upgrading then only postpones the problem; the same load continues on more expensive hardware.
Our second observation is that the gap between analytics and server graphs goes unquestioned. Analytics largely excludes bot traffic while the server serves every request. A large difference between those two sources is a strong signal on its own and deserves investigation.
Third, log management is the cheapest win available. Outages caused by full disks are the easiest kind to prevent. Automatic rotation plus a usage alert — the threshold covered in our monitoring article — closes that scenario entirely.
Investigation checklist
- Server request counts compared with analytics sessions.
- Highest-volume user agents listed from access logs.
- High-volume requests from single IPs identified.
- Scheduled tasks listed with run times and durations noted.
- Disk usage examined by directory (logs, cache, backups, uploads).
- Log rotation configured and working.
- Slow query log reviewed over at least a week.
- Cache rules verified; price and stock not served from cache.
- Most-requested dynamic addresses identified.
- Backup schedule moved outside peak hours.
Frequently asked questions
Can I just block every bot?
No. Blocking search engine crawlers means disappearing from search results, and blocking AI crawlers costs similar visibility. Rather than blocking indiscriminately, limit unwanted automated traffic by behaviour.
How do I investigate this on shared hosting?
Panels usually provide access logs, disk usage detail and resource graphs. Those three suffice to identify four of the six sources. Slow query analysis may require your provider's help.
Do cron jobs really produce that much load?
They can, particularly when all are scheduled for the same minute, or when a job runs longer than expected and overlaps the next run. The same work then runs several times concurrently and load multiplies. Recording run durations makes this visible immediately.
Why does a full disk stop the site entirely?
Because the database and application can no longer write temporary files, operations fail outright. The site does not slow — it errors. Disk usage is therefore among the highest-priority metrics to monitor.
Is caching suitable for every page?
No. Pages with sessions, the cart, the checkout step and the admin panel must not be cached. On pages carrying price or stock data, keep cache lifetimes short or purge on update.
Conclusion
When server load rises, the first question should not be "should I move up a plan" but "who is producing this load". Examining the six sources takes a few hours and usually delivers more than a hardware upgrade.
If you would like us to analyse your server load, get in touch. You can review our server management, VPS server and fast hosting solutions.