Collect profiles from other threads.

[tmc]

Different serving environments have different thread utilizations -- only collecting from the main thread doesn't capture all workloads correctly.

[ar-qun]

This would be good feature to have. Recently we had worked on a Flask app and were confused as to why none of our code showed up.

[lunalucadou]

Thirding this feature request. After upgrading a Google Cloud Function from 1st gen to 2nd gen, profiling no longer works because gen 2 Cloud Functions are concurrent by default. Although we're not sure how much extra performance it provides under heavy load due to the GIL, it does keep functions hot for longer, thus reducing initialization times, so we don't have a reason to disable it, either.

We shouldn't have to choose between concurrency and profiling, especially when the Java and Go profilers don't have this limitation. [Unknown - No Python thread state] is where all our code is being executed, and that isn't a very helpful metric.

[aabmass]

I'd like to clarify what is currently supported today:

• The CPU profiler already supports profiling multiple python threads. Any non-python threads will be grouped together into a single frame with [Unknown - No Python thread state].
• The Wall time profiler only profiles the python main thread

  cloud-profiler-python/googlecloudprofiler/pythonprofiler.py

  Lines 32 to 37 in 9952717

  	class WallProfiler:
  	"""Python2-compatible implementation for Wall time profiler.
  	This Wall profiler avoids dependency on any native code by using
  	Python signal module. It only profiles the main thread. It has the following
  	limitations:

[aabmass]

[Unknown - No Python thread state] is where all our code is being executed, and that isn't a very helpful metric.

@lunalucadou it looks like you're using the CPU profiler, so all of your python code running in any python thread should be showing in the profile. The frame with [Unknown - No Python thread state] is for threads that don't have a PyThreadState (non-python threads). Are you using a native extension that creates its own threads? What version of python are you running?

[lunalucadou]

@aabmass Python 3.11. We don't explicitly use any concurrency; any threads are just whatever Cloud Functions gen 2 does by default.

[aabmass]

I don't think this is the case but it would be interesting to know if profiles look different in Python 3.10. I made some changes to support 3.11 but i don't think it would cause this.

We also don't technically support Cloud Functions, meaning I can't provide much support if its something specific to that platform. https://cloud.google.com/profiler/docs/about-profiler#environment_and_languages

[aabmass]

@lunalucadou I tried it out and see the same issue with Cloud Functions gen 2 regardless of concurrency. I think what's happening is that gen 2 is using a post-fork server (I believe gunicorn) to fork worker processes and profiler is only set up on the original process. Similar to what is described for uWSGI in the docs https://cloud.google.com/profiler/docs/profiling-python#known-issues.

I would reach out to Cloud Functions team or the functions-framework-python repo and see if they can make a general recommendation (Cloud Profiler team can't provide any support on this platform). One thing I tried is using os.register_at_fork() and that fixed the issue:

GCF source code

import functions_framework
import os

def setup_profiler():
    service_version = "1.0.1-postfork8"
    print("======== Setting up profiler in postfork process for {}".format(service_version))
    import googlecloudprofiler

    # Profiler initialization. It starts a daemon thread which continuously
    # collects and uploads profiles. Best done as early as possible.
    try:
        googlecloudprofiler.start(
            service="hello-profiler",
            service_version=service_version,
            # verbose is the logging level. 0-error, 1-warning, 2-info,
            # 3-debug. It defaults to 0 (error) if not set.
            verbose=3,
            # project_id must be set if not running on GCP.
            # project_id='my-project-id',
        )
    except (ValueError, NotImplementedError) as exc:
        print(exc)  # Handle errors here

# initialize profiler in each subprocess after it forks
os.register_at_fork(after_in_child=setup_profiler)

@functions_framework.http
def hello_http(request):
    """HTTP Cloud Function.
    Args:
        request (flask.Request): The request object.
        <https://flask.palletsprojects.com/en/1.1.x/api/#incoming-request-data>
    Returns:
        The response text, or any set of values that can be turned into a
        Response object using `make_response`
        <https://flask.palletsprojects.com/en/1.1.x/api/#flask.make_response>.
    """
    request_json = request.get_json(silent=True)
    request_args = request.args

    if request_json and 'name' in request_json:
        name = request_json['name']
    elif request_args and 'name' in request_args:
        name = request_args['name']
    else:
        name = 'World'
    return 'Hello {}! {}'.format(name, myotherfunc())

def myotherfunc():
    def innerfunc():
        return 1 + 1
    
    out = 1
    for i in range(10 ** 4):
        out *= innerfunc() * innerfunc()
    return out % 1000000000000000

Again the Cloud Functions team might have a better recommendation and this needs some tweaking to set up in the root process only once.