Adventures in Machine Learning

Building a Python Web App with Bottle SQLAlchemy and Tweepy: A Tutorial

Building a Python Web App with Bottle, SQLAlchemy and Tweepy

Have you ever fancied building a web app from scratch, using your Python skills? If so, then this tutorial is for you.

In this article, we will go through the process of creating a web app from scratch, using a range of technologies including Bottle, SQLAlchemy, and Tweepy. We will start by setting up the project and environment, then move onto querying the Twitter API, making a simple web app with Bottle, and adding tests with pytest.

Finally, we will go over how to use Better Code Hub to improve the maintainability and modularity of your code.

Project Setup

Before we start writing our web app, we need to set up our project environment. This involves creating production and test databases in Postgres, storing configuration in environment variables, and cloning the repository and installing dependencies using virtual environment.

By doing this, we are ensuring a secure and robust foundation for our web app.

Query the Twitter API

Once our environment is set up, we can move onto querying the Twitter API using Tweepy. We will go over how to use pagination to retrieve a large number of tweets, how to search for hashtags, and how to use Counter to count the occurrence of words in tweets.

By the end of this section, you will have a solid understanding of how to extract data from Twitter using Python.

Make a Simple Web App with Bottle

Next, we will make a simple web app using the Bottle framework. We will go over routing, template engines, and static files.

In no time, you will have a web app that displays tweets containing a particular hashtag, all beautifully displayed on a webpage.

Add Tests with pytest

Once our web app is up and running, it’s time to add tests to our codebase. We will go over how to use pytest to write tests for both static data and the test database.

By doing so, we’ll ensure that our code is robust and reliable.

Better Code Hub

Now that we have a working web app with tests, it’s time to ensure that the code is maintainable and modular. We will use Better Code Hub to analyze our codebase and provide suggestions for improvement. We will go over refactorings that you can make to your codebase to make it more modular and maintainable.

Conclusion and Learning

By the end of this tutorial, you will have learned how to create a complete web app from scratch using Python. We’ve covered various topics, including querying the Twitter API, creating a web app using Bottle, testing your code, and improving maintainability using Better Code Hub. You will have a solid understanding of how to use Bottle, SQLAlchemy, Tweepy, and pytest.

You’ll be well on your way to building more complex and robust web apps using Python.

In summary, building a Python web app using these technologies can be a rewarding and enriching experience.

Once set up, you should be able to extract data from Twitter, store it in a database, create a web app to display that data, and ensure that your code is reliable using tests and Better Code Hub. We hope that you’ve found this tutorial informative and wish you the best of luck in your web app development journey.

Querying the Twitter API

Twitter is one of the most popular social media platforms in the world. With millions of tweets being posted every day, it’s an excellent source of data for businesses and researchers.

In this section, we’ll explore how to query the Twitter API using Python’s Tweepy library, which will help us import tweets and extract hashtags.

Import Tweets with Tweepy

Tweepy is a Python library that enables developers to access Twitter API in their applications. To use Tweepy, you’ll need to sign up for a Twitter Developer account and create an app.

Once you have your app’s credentials, you can set up Tweepy to access your Twitter account.

The first thing we’ll do is set up pagination, which will help us retrieve a large number of tweets.

This is done by setting a limit on the number of tweets to return per page and using the Cursor() class to iterate through all the pages of tweets.

import tweepy
# Twitter API credentials
consumer_key = "your_consumer_key"
consumer_secret = "your_consumer_secret"
access_key = "your_access_token"
access_secret = "your_access_secret"
# API authorisation 
auth = tweepy.OAuthHandler(consumer_key, consumer_secret)
auth.set_access_token(access_key, access_secret)
# Construct API instance
api = tweepy.API(auth, wait_on_rate_limit=True,
                 wait_on_rate_limit_notify=True)
# Maximum number of tweets to import
max_tweets = 5000
tweets = []
# Loop through pages
for i, tweet in enumerate(tweepy.Cursor(api.search_tweets,
                                        q='#Python').items(max_tweets)):
    tweets.append(tweet.text)
    if i % 100 == 0:
        print(f"Imported {i} tweets")

In the above example, we are importing tweets containing the hashtag #Python. The Cursor() class is being used to iterate through all the pages of tweets, and we’re only importing 5000 tweets.

Extract Hashtags with Regex and Collections

Once we’ve imported our tweets, we may want to extract the hashtags from them. To do this, we can use the Python re module, which enables us to use regular expressions to find patterns in strings.

import re
from collections import Counter
hashtags = []
for tweet in tweets:
    hashtag_list = re.findall(r'#w+', tweet)
    if hashtag_list:
        hashtags += hashtag_list
# Count the occurrence of each hashtag
hashtag_count = Counter(hashtags)

In the above example, we are using the findall() method of the re module to search each tweet for hashtags. The Counter() method from the collections module is then used to count the occurrence of each hashtag.

Building a Web App with Bottle

Now that we’ve extracted data from Twitter, it’s time to build a web app to display that data. In this section, we’ll explore Bottle, a micro-framework for Python web development.

We’ll create routes, use templates to display data, and work with static files.

Bottle is a lightweight Python web framework, often compared to Flask. It’s a great choice for small web applications that don’t require a full-blown framework.

Bottle acts as a WSGI (Web Server Gateway Interface) application, which means it can run on any server that supports WSGI.

To use Bottle, you’ll need to install it using pip.

You can then import the Bottle class and create a new instance.

from bottle import Bottle
app = Bottle()

Creating Routes and Templates

The next step is to create routes for our web app. A route is a URL that the user can access to display a specific page of your app.

To create routes, we’ll use the @app.route() decorator and define a function that returns the HTML to be displayed.

from bottle import route, run, template
@app.route('/')
def index():
  return template('index.tpl')

In the above example, we are defining a route for the home page of our app, index().

This function returns the HTML generated by the index.tpl file, which is a template located in the views folder of our app.

Working with Static Files

Static files are those that do not change during the execution of your web app, such as CSS or JavaScript files. To serve these files, we’ll use the static_file() function.

from bottle import static_file, run
@app.route('/static/')
def send_static(filename):
    return static_file(filename, root='static')
if __name__ == '__main__':
    run(host='localhost', port=8080, debug=True)

In the above example, we’re defining a route that serves the files in the static folder of our app. We’re using the static_file() function to serve the files, and the run() method of the Bottle class to run our app.

Deployment and Debugging

Once we’ve completed our web app, it’s time to deploy it to a server. To do this, we can use a hosting provider like Heroku or AWS.

We’ll need to set up some environment variables and modify our code to ensure that it runs on the server correctly.

Debugging can be challenging, especially when working with web apps.

We can use the debug() method of the Bottle class to run our app in debug mode, which will help us identify any issues.

if __name__ == '__main__':
    app.run(host='localhost', port=8080, debug=True)

In the above example, we’re adding debug=True to the run() method.

This will run our app in debug mode, which displays detailed error messages whenever an error occurs.

In summary, Bottle is a micro-framework for Python web development that offers a lightweight and fast way to create and deploy web apps.

We’ve seen how to create routes, templates, and serve static files. We also covered deployment and debugging.

Small, efficient, and ideal for small-scale web applications, Bottle provides an excellent foundation for your Python web app development.

Adding Tests with pytest

Testing is an essential part of developing software. Testing your code ensures that it works as expected, and it provides you with the confidence to make changes and improvements to your codebase.

In this section, we’ll explore how to use pytest, a popular testing framework for Python. We’ll cover fixtures and testing the database.

Also, we’ll go over testing hashtags with multiline strings.

Pytest is a testing framework for Python that makes testing your code easy and straightforward. To get started, you’ll need to install pytest using pip.

Then, you can write tests in Python using the assert statement.

def test_addition():
    assert 1 + 1 == 2

In the above example, we’re testing that 1 + 1 equals 2.

If the assertion fails, the test will fail and output an error message.

Testing the DB with Fixtures

In some applications, you may need to work with a database. To test your database queries, you can use fixtures in pytest.

A fixture is a piece of code that sets up some prerequisites for a test.

import pytest
from sqlalchemy import create_engine
from sqlalchemy.orm import sessionmaker

from app import models

@pytest.fixture(scope='module')
def test_db():
    engine = create_engine('postgresql://user:password@host/db_name')
    Session = sessionmaker(bind=engine)
    session = Session()
    models.Base.metadata.create_all(engine)
    yield session
    models.Base.metadata.drop_all(engine)
    session.close()

In the above example, we’re defining a new fixture, test_db(). This fixture sets up a connection to a PostgreSQL database with the credentials user and password and the database name db_name.

It then creates a new session and tests the database model’s creation. Finally, it drops the tables and closes the session.

We can then use this fixture in our tests by passing it as an argument to the test function.

def test_get_user(test_db):
    user = models.User(username='test_user')
    test_db.add(user)
    test_db.commit()
    assert models.User.get_by_username('test_user').id == user.id

In the above example, we’re using the test_db fixture to test the User model’s get_by_username() method.

We’re adding a new user to the database, committing the transaction, and then testing that we can retrieve the user by their username.

Testing Hashtags with Multiline Strings

When parsing social media data, it’s essential to test that your code can handle multiple hashtags per tweet. We can do this using multiline strings in our tests.

A multiline string is a string that spans multiple lines.

def test_extract_hashtags():
    tweet = """
    Here is an example tweet with 
    #Python and #programming hashtags
    """
    expected_hashtags = ['Python', 'programming']
    assert extract_hashtags(tweet) == expected_hashtags

In the above example, we’re testing a function that extracts hashtags from a tweet.

We’re using a multiline string to simulate a tweet with two hashtags. We then pass this tweet to our extract_hashtags() function and compare the expected hashtags to the extracted hashtags.

Better Code Hub

Better Code Hub is a tool that analyzes your codebase and provides suggestions for improvement. It uses a set of rules to identify potential issues with the modularity, maintainability, and security of your code.

To use Better Code Hub, you’ll need to create an account and connect your repository. The tool will analyze your codebase and output a report with suggestions for improvement.

Each suggestion will come with a brief explanation and a severity rating.

Refactorings for Improved Modularity

One of the most important things that Better Code Hub looks for is modularity. Modularity refers to how well your codebase is organized into reusable and independent modules.

Here are some tips on how to improve modularity in your codebase:

  1. Keep your functions small and focused
  2. Use descriptive variable and function names
  3. Avoid global variables when possible
  4. Use object-oriented programming when it makes sense

You can also use the tips directory generated by Better Code Hub to get more specific suggestions for your codebase.

Achieving a Perfect Score

Once you’ve identified and made the necessary improvements to your codebase, you can aim for a perfect score on Better Code Hub. A perfect score means that your codebase meets all of the tool’s modularity, maintainability, and security standards.

To achieve a perfect score, you’ll need to follow best practices for writing high-quality Python code. These include using virtual environments, adhering to PEP 8 guidelines, and testing your code thoroughly with pytest.

In summary, pytest is a powerful testing framework for Python that can help ensure the reliability of your codebase.

Better Code Hub is an excellent tool for analyzing your codebase and providing suggestions for improvement. By following best practices for modularity, maintainability, and security, you can achieve a perfect score and write clean, high-quality Python code.

Conclusion and Learning

In this tutorial, we’ve covered a range of topics related to building a Python web app using Bottle, SQLAlchemy, and Tweepy. We started by setting up the project and environment, querying the Twitter API, building the web app with Bottle, adding tests with pytest, and using Better Code Hub to improve code quality.

Overview of the Completed App and Modules Used

The completed app allows users to search for tweets containing a particular hashtag. It uses Tweepy to import data from Twitter, SQLAlchemy to store the data in a database, and Bottle to display the data on a webpage.

We’ve also covered testing our code using pytest, ensuring the reliability and robustness of our app using database fixtures and multiline string tests. Finally, we discussed using Better Code Hub to provide suggestions for improving the modularity and maintainability of our codebase.

Lessons

  • Building a web app from scratch using Python can be rewarding and enriching.
  • Using a micro-framework like Bottle can help to simplify the development process.
  • Testing your code is essential for ensuring its quality and reliability.
  • Tools like Better Code Hub can help to identify areas for improvement in your codebase.

Popular Posts