ChatGPT deep dive: Will AI be writing production grade software any time soon?

ChatGPT, an AI chat assistant from OpenAI, took the world by storm when it was first released at the end of November 2022. Powered by an advanced language model, GPT-3, we saw a myriad of wonderful use cases sweep the internet: content generation, money making ideas and even writing code.

Recently, ChatGPT has been upgraded. GPT-4 is backed by 100 trillion parameters compared to the 175 billion of GPT-3. So with this added power, how good is it actually at writing software?

Show me what you got ChatGPT

Let's start things off with a Javascript and React Native question. Writing boilerplate code for forms is a necessary but rather mundane task in the world of software, so let's see if ChatGPT can help us out. We're even going to ask it to use some open source libraries that will make our lives easier in future development. We will give it the following prompt:

produce a react native form, using formik and yup validation, with email and password inputs and a submit button that says Login

The resulting code is actually pretty decent. It's even given us error states without even asking for them:

import React from 'react';
import { View, TextInput, StyleSheet, Button } from 'react-native';
import { Formik } from 'formik';
import * as Yup from 'yup';

const validationSchema = Yup.object().shape({
  email: Yup.string().email().required('Email is required'),
  password: Yup.string()
    .min(8, 'Password must be at least 8 characters')
    .required('Password is required'),
});

export default function LoginForm() {
  return (
    <Formik
      initialValues={{ email: '', password: '' }}
      validationSchema={validationSchema}
      onSubmit={values => console.log(values)}
    >
      {({ handleChange, handleBlur, handleSubmit, values, errors, touched }) => (
        <View style={styles.container}>
          <TextInput
            style={styles.input}
            placeholder="Email"
            onChangeText={handleChange('email')}
            onBlur={handleBlur('email')}
            value={values.email}
            keyboardType="email-address"
            autoCapitalize="none"
            autoCorrect={false}
          />
          {errors.email && touched.email && <Text style={styles.error}>{errors.email}</Text>}
          <TextInput
            style={styles.input}
            placeholder="Password"
            onChangeText={handleChange('password')}
            onBlur={handleBlur('password')}
            value={values.password}
            secureTextEntry
          />
          {errors.password && touched.password && <Text style={styles.error}>{errors.password}</Text>}
          <Button title="Login" onPress={handleSubmit} />
        </View>
      )}
    </Formik>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    alignItems: 'center',
    justifyContent: 'center',
    marginHorizontal: 20,
  },
  input: {
    height: 50,
    width: '100%',
    borderWidth: 1,
    borderColor: 'gray',
    paddingHorizontal: 10,
    marginVertical: 10,
  },
  error: {
    color: 'red',
  },
});

Hold on a minute, we forgot something. We love type safety at Logicful so let's ask ChatGPT to expand on this a little bit and add Typescript support to our code example:

import React from 'react';
import { View, TextInput, StyleSheet, Button, Text } from 'react-native';
import { Formik, FormikProps } from 'formik';
import * as Yup from 'yup';

interface LoginFormValues {
  email: string;
  password: string;
}

const validationSchema = Yup.object().shape({
  email: Yup.string().email().required('Email is required'),
  password: Yup.string()
    .min(8, 'Password must be at least 8 characters')
    .required('Password is required'),
});

export default function LoginForm() {
  const handleSubmit = (values: LoginFormValues) => {
    console.log(values);
  };

  return (
    <Formik
      initialValues={{ email: '', password: '' }}
      validationSchema={validationSchema}
      onSubmit={handleSubmit}
    >
      {(props: FormikProps<LoginFormValues>) => (
        <View style={styles.container}>
          <TextInput
            style={styles.input}
            placeholder="Email"
            onChangeText={props.handleChange('email')}
            onBlur={props.handleBlur('email')}
            value={props.values.email}
            keyboardType="email-address"
            autoCapitalize="none"
            autoCorrect={false}
          />
          {props.errors.email && props.touched.email && <Text style={styles.error}>{props.errors.email}</Text>}
          <TextInput
            style={styles.input}
            placeholder="Password"
            onChangeText={props.handleChange('password')}
            onBlur={props.handleBlur('password')}
            value={props.values.password}
            secureTextEntry
          />
          {props.errors.password && props.touched.password && <Text style={styles.error}>{props.errors.password}</Text>}
          <Button title="Login" onPress={props.handleSubmit} />
        </View>
      )}
    </Formik>
  );
}

const styles = StyleSheet.create({
  container: {
    flex: 1,
    alignItems: 'center',
    justifyContent: 'center',
    marginHorizontal: 20,
  },
  input: {
    height: 50,
    width: '100%',
    borderWidth: 1,
    borderColor: 'gray',
    paddingHorizontal: 10,
    marginVertical: 10,
  },
  error: {
    color: 'red',
  },
});

Great! We've got a little bit of work to do - namely adding in logic for what we want to actually do when the form is submitted. Most of our boilerplate code is written though. We've just saved ~10 minutes with the help of our digital assistant. Over the course of a whole project this could amount to savings of entire days if we outsource our boilerplate code production.

Well played ChatGPT, now let's log our user in

So ChatGPT and its 100 trillion parameters are pretty good at taking instructions to produce some frontend code for our mobile application written in React Native. What if we need it to troubleshoot something we are don't quite know how to do ourselves though? Can we start trusting the knowledge it has to write actual business logic in our application?

Let's change tack slightly and head back to the world of web applications and ask for a login form. This time we want to actually log our user in and have our authentication managed by supabase. We will also be using NextJS, so let's throw another prompt at our AI friend:

In nextjs, how do I create a login form that works with the supabase sdk?

The response for this prompt is quite long so I have amalgamated the code and instructions provided:

// 1. Install the modules

npm install @supabase/supabase-js

// 2. Create a supabaseClient.js in the root of our project

import { createClient } from '@supabase/supabase-js';

const supabaseUrl = 'https://yourproject.supabase.co';
const supabaseKey = 'yourpublicapikey';

export const supabase = createClient(supabaseUrl, supabaseKey);

// 3. Create a page in the pages directory of your Next project called login.js

import { useState } from 'react';
import { supabase } from '../supabaseClient';

export default function Login() {
  const [email, setEmail] = useState('');
  const [password, setPassword] = useState('');

  const handleLogin = async (event) => {
    event.preventDefault();

    const { user, error } = await supabase.auth.signIn({
      email,
      password,
    });

    if (error) {
      alert(error.message);
    } else {
      alert(`Logged in as ${user.email}`);
    }
  };

  return (
    <form onSubmit={handleLogin}>
      <label htmlFor="email">Email:</label>
      <input
        type="email"
        id="email"
        name="email"
        required
        value={email}
        onChange={(event) => setEmail(event.target.value)}
      />
      <br />
      <label htmlFor="password">Password:</label>
      <input
        type="password"
        id="password"
        name="password"
        required
        value={password}
        onChange={(event) => setPassword(event.target.value)}
      />
      <br />
      <button type="submit">Login</button>
    </form>
  );
}

Again, our AI helper has done a pretty stellar job of providing the basic functionality we need. It has provided an NPM package that we need and understood the context of how to integrate this into our project. However here we begin to see the limitations of the AI. I know that this is not the most efficient solution for handling authentication with Supabase in a NextJS project. I can read their technical documentation and I know that they actually have a very helpful library specifically designed for integrating Supabase authentication with NextJS and handling user sessions.

Let's give our AI friend the benefit of the doubt though. We have to be explicit about these things and ultimately the solution provided would work. I know from reading the Supabase documentation that making use of their NextJS library will be hugely helpful to our project as it grows, so let's give it a bit of a nudge:

Show me how to use the supabase nextjs auth helpers library

// 1. Install the library

npm install @supabase/supabase-nextjs-auth

// 2. Create a new file called supabase.js in your project's root

import { createClient } from '@supabase/supabase-js';

export const supabase = createClient(
  'https://yourproject.supabase.co',
  'yourpublicapikey'
);

// 3. In your _app.js file, wrap your Component with the SupabaseAuthProvider component. This component will provide the Supabase client and user authentication state to your application:

import { supabase } from '../supabase';
import { SupabaseAuthProvider } from '@supabase/supabase-nextjs-auth';

function MyApp({ Component, pageProps }) {
  return (
    <SupabaseAuthProvider supabaseClient={supabase}>
      <Component {...pageProps} />
    </SupabaseAuthProvider>
  );
}

export default MyApp;

// 4. You can use the useUser hook to get the current authenticated user:

import { useUser } from '@supabase/supabase-nextjs-auth';

export default function Profile() {
  const { user } = useUser();

  if (!user) {
    return <div>Loading...</div>;
  }

  return <div>Welcome, {user.email}!</div>;
}

// 5. An example of using the ProtectedRoute component

import { ProtectedRoute } from '@supabase/supabase-nextjs-auth';

export default function Dashboard() {
  return (
    <ProtectedRoute>
      <div>This is the dashboard page</div>
    </ProtectedRoute>
  );
}

These are just a few examples of how you can use the @supabase/supabase-nextjs-auth library to add authentication to your Next.js application with Supabase. The library also provides other pre-built components and utilities that you can use to customize your authentication flow. You can find more information in the official documentation: https://supabase.io/docs/guides/auth-nextjs-auth-helper.

So we get a great explanation of how to implement this library in our existing application. We even get a link to the documentation to read more, thank you AI! There's just one small problem with the example provided here...

...A large portion of what it has returned is a lie. ChatGPT, rather than tell us that it doesn't actually know, has lied to us. That package does not exist. The API of the package, even if it were the correct one, is not correct. That link at the end to the documentation for this package it claims to be using takes us to a 404 page. Ultimately, there is just no way that what ChatGPT has provided will work.

ChatGPT is a great tool, but it cannot access the internet and its knowledge is not limitless. The AI can only utilise explicit information that it has been trained with. Either the information it has provided to us is out of date, or it is trying to piece together data points that it does have in order to confidently give us a result that looks correct.

To the untrained eye this looks like perfectly serviceable code and we will still continue to hear cries that the days of software engineers are numbered in favour of savvy growth hackers and their AI henchmen. That day may come eventually, but it isn't going to happen for a little while yet. At least ChatGPT has the good manners to admit when it is wrong:

Conclusion

This may read a little bit like an old man screaming at the marvels of modern technology, but I do not mean it to. GPT-4 and it's chat interface, ChatGPT, is a fantastic piece of technology. It's a great time saver in the right hands for a myriad of tasks - including writing code for you or giving you a good jumping off point for a project. The first example we used in this article is a fantastic use case for having an AI helper handle the tedious parts of programming.

Code is just one piece of the puzzle though. We also need to be able to deploy our software - likely onto cloud based infrastructure. ChatGPT can't access the internet, so while it could give us advice on how to do this, one would still need to know what they were doing to get our code deployed into production.

In the hands of a good software engineer it's a great tool. In the hands of an untrained or poor engineer, at best it could be frustrating. At worst it could be disastrous and cost your business money.

If you want a group of professionals, with the help of AI, to solve your software needs then look no further, get in contact.