Troubleshooting
FAQs
Even well-designed systems encounter issues. This guide walks you through common problems you might face with NoBS Python and explains how to diagnose and fix them.
Deployment Fails to Start
You run nobs deploy and the command completes, but when you check the dashboard, your services show as "Failed" or keep restarting indefinitely. This usually means something in your code is preventing the container from starting successfully.
Start by viewing the logs for the failing service. Click into the project, find the troubled component, and open its log viewer. Look for Python exceptions, import errors, or startup failures. Common culprits include:
Missing dependencies. If you added a new package to your code but forgot to include it in pyproject.toml or requirements.txt, the container won't have that library available. The logs will show an ImportError or ModuleNotFoundError. Fix this by adding the dependency and redeploying.
Configuration errors. If your application expects a secret or environment variable that isn't set, it might crash during startup. Pydantic Settings will raise a ValidationError if required fields are missing. Check the secrets management page to ensure all required values are configured for your environment.
Database connection failures. If your application tries to connect to a database immediately on startup and the connection fails, the container exits with an error. This can happen if database credentials are wrong, the database instance isn't provisioned yet, or there's a network connectivity issue. Verify your database connection string in the secrets configuration.
If you can't identify the issue from logs alone, try running nobs up locally to reproduce the problem. Local deployment often makes errors more visible and easier to debug.
Services Are Slow or Unresponsive
Your application deployed successfully, but requests are taking much longer than expected or timing out entirely. This often points to resource constraints or inefficient code.
Check the resource usage metrics in the dashboard. If CPU or memory is maxed out, your service might not have enough allocated resources. Edit your project.py to increase the Compute limits:
server=FastAPIApp(
app,
compute=Compute(
mvcpu_limit=2000, # Increase from 1000
mb_memory_limit=2048, # Increase from 1024
)
)
Redeploy after making this change. Higher resource limits give your application more room to breathe under load.
If resource usage looks fine but performance is still poor, examine your application's database queries. Slow queries are a common bottleneck. Look for N+1 query patterns, missing indexes, or full table scans. Use database profiling tools to identify which queries are taking the longest.
Network latency can also contribute to slowness. If your application calls external APIs frequently, consider implementing caching or batching requests to reduce round-trips.
Worker Queue Is Growing
You check your worker status and see the queue depth increasing steadily. Messages are piling up faster than the worker can process them. This creates a backlog that leads to delays and potential data loss if the queue fills completely.
First, determine if the worker itself is healthy. Check the worker logs for errors or exceptions. If every message fails to process, the queue grows because nothing gets removed. Fix any bugs in the worker function and redeploy.
If the worker is processing messages successfully but just can't keep up with the volume, you need more throughput. Increase the max_scale parameter to allow multiple worker instances to run in parallel:
project = Project(
workers=[priority_queue],
priority_queue_worker=Worker(
"priority_queue",
max_scale=5, # Allow up to 5 worker instances
)
)
This lets NoBS Python spin up additional workers when the queue depth grows, distributing the load across multiple containers.
You can also optimize the worker function itself. If processing each message takes a long time, look for opportunities to parallelize work, cache expensive computations, or reduce unnecessary I/O.
Secrets Not Being Injected
Your application is deployed, but when it tries to read a secret value, it gets None or raises a validation error. This means the secret isn't being injected properly.
Verify that you've declared the secret in your project definition. Secrets must be explicitly listed in either shared_secrets or in the component's secrets parameter:
project = Project(
name="my-project",
shared_secrets=[SharedSecrets], # Available to all components
server=FastAPIApp(
app,
secrets=[StripeConfig], # Only available to this app
)
)
Next, check that the secret values are actually configured. Go to the Secrets section in the dashboard, select your environment, and confirm that all required fields have values. Empty fields won't be injected.
Finally, ensure your Pydantic Settings class is reading from environment variables correctly. By default, Pydantic converts field names to uppercase and looks for them in os.environ. A field named openai_api_key will look for OPENAI_API_KEY in the environment.
Job Fails with Timeout
You trigger a scheduled job or run it manually, and it fails with a timeout error. Jobs have a maximum execution time, typically 15 minutes by default. If your job exceeds this limit, it gets killed.
For long-running tasks, increase the timeout in your job definition:
long_task=Job(
process_large_dataset,
timeout=3600, # Allow 1 hour (in seconds)
)
If the job still times out, consider breaking it into smaller chunks. Instead of processing an entire dataset at once, process batches and track progress. This makes the job more resilient and easier to debug.
Alternatively, move extremely long workloads to a worker instead of a job. Workers don't have strict time limits and can process tasks that take hours if needed.
Can't Access Deployed Application
You deployed successfully and see your service marked as "Running," but when you try to access its URL, you get a 404 or connection refused error.
First, verify the port configuration. Your NetworkApp must specify the correct port that your application listens on:
custom_app=NetworkApp(
command=["python", "server.py"],
port=8000, # Must match the port your server binds to
)
If the port is correct, check the health check. If your application defines a health check endpoint but that endpoint returns errors, NoBS Python might mark the service as unhealthy and refuse to route traffic to it. Fix the health check endpoint or remove it temporarily to see if traffic starts flowing.
Domain name configuration can also cause access issues. If you specified custom domains but haven't pointed DNS records at the NoBS infrastructure, requests to those domains will fail. Verify that your DNS is configured correctly and wait for propagation (which can take up to 48 hours in some cases).
Local Environment Won't Start
You run nobs up but containers fail to start or exit immediately. This is usually a Docker-related issue.
Ensure Docker is actually running on your machine. Run docker ps to confirm Docker responds. If it doesn't, start Docker Desktop or your container runtime.
Volume mount issues can also prevent startup. If NoBS Python can't access your source code directory due to permission problems, containers won't launch. Make sure your project directory isn't inside a restricted filesystem location.
Database Connection Strings Are Wrong
Your application tries to connect to PostgreSQL or Redis, but the connection fails with authentication or host not found errors. This usually means the connection string secret is incorrect.
For NoBS-provisioned databases, the connection string is automatically generated and stored as a secret. Don't edit these manually—if you do, you'll break the connection. Instead, let NoBS Python regenerate them by deleting the old secret and redeploying.
If you're using an external database (one you provisioned outside NoBS Python), double-check the connection string format. PostgreSQL uses postgresql://user:password@host:port/database. Redis uses redis://host:port. Typos in these URLs cause connection failures.
Need More Help
If you've tried the above solutions and still can't resolve your issue, reach out for support. The AI chat assistant in the dashboard can often diagnose problems by analyzing your logs and configuration. Ask it specific questions like "Why did my deployment fail?" or "Why is my worker not processing messages?"
For issues requiring human intervention, contact support through the dashboard's support ticket system or email the NoBS Python team. Include relevant logs, error messages, and a description of what you were trying to do when the problem occurred. The more context you provide, the faster the team can help you.