Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Sometimes 403 Forbidden server response when websocket message call ajax update with identity store #25141

Open
Dev-ALPM opened this issue Sep 11, 2024 · 12 comments · May be fixed by #25290
Open

Comments

@Dev-ALPM
Copy link

Environment Details

  • Eclipse GlassFish 7.0.18-SNAPSHOT (commit: b4bd2e2)
  • JDK version: 21.0.3
  • OS: Windows 10

Problem Description

Sometimes server return 403 forbidden

I add RealmAdapter log:

Normal trace:
RealmAdapter.findPrincipalWrapper
principal:jakarta.security.enterprise.CallerPrincipal@31614fef
sessionPrincipal:jakarta.security.enterprise.CallerPrincipal@31614fef|#]
RealmAdapter.findPrincipalWrapper
principal:jakarta.security.enterprise.CallerPrincipal@41e82e26
sessionPrincipal:jakarta.security.enterprise.CallerPrincipal@41e82e26|#]
sessionBean.getText(): TEST1|#]
sessionBean.getText(): TEST2|#]

When 403 forbidden:
RealmAdapter.findPrincipalWrapper
principal:jakarta.security.enterprise.CallerPrincipal@41e82e26
sessionPrincipal:jakarta.security.enterprise.CallerPrincipal@31614fef|#]
RealmAdapter.findPrincipalWrapper
principal:jakarta.security.enterprise.CallerPrincipal@31614fef
sessionPrincipal:jakarta.security.enterprise.CallerPrincipal@31614fef|#]
RealmAdapter.validate sessionSubject == null|#]
sessionBean.getText(): TEST1|#]

Server return 403 for TEST2,
In fonction RealmAdapter.findPrincipalWrapper caller is TEST2 and sessionPrincipal is TEST1

Steps to reproduce

TestSession.zip

Compile TestSession project
Deploy TestSession

Open chrome explorer and developper tools, call http://127.0.0.1:8080/test/
login with TEST1 / TEST1

Open edge explorer and developper tools, call http://127.0.0.1:8080/test/
login with TEST2 / TEST2

Sometimes server return 403 forbidden

Impact of Issue

Client not refresh when server return 403

403-forbidden

@arjantijms
Copy link
Contributor

Thanks for the report. A first question; does this happen with any two browsers, or specifically with chrome and edge?

What is the approximate frequency of sometimes failing? Eg in the order of 1 in 100 or 1 in 10 or so?

@Dev-ALPM
Copy link
Author

It's not specifically Chrome, I also have an error with Firefox

I've test with WildFly 33.0.1, and not problem
testSession-WildFly.zip

For test :
start wildfly standalone with -c standalone-full.xml parameter
Disable Integrated JASPI under management console : Configuration / Subsytem / Web / Application Security Domain
mvn install wildfly:deploy

@Dev-ALPM
Copy link
Author

After using glassfish since 2006, I was forced to adapt my projects for WildFly.

I did not have time to do further research to target the code of the server concerned.

For me, the problem happens when the request uses a different thread from the http pool between two calls.

The TestSession project allows you to reproduce the problem quickly with opening two browsers.

@OndroMih
Copy link
Contributor

The same issue also reported here: https://stackoverflow.com/questions/79070073/response-403-forbidden-at-simultaneously-ajax-request-using-jakarta-ee-and-glass

@Isaacrgz
Copy link

Environment Details

  • Eclipse GlassFish version:
  • 7.0.16
  • 7.0.18
  • JDK version:
  • 17.0.11
  • OS:
  • Windows 11 Professional
  • Windows Server 2012
  • Request target:
  • Servlet
  • RESTful service
  • Browser:
  • Google Chrome Dev
  • Edge
  • Security checking
  • request.isUserInRole
  • securityContext.isCallerInRole

Problem Description

In my case, it happend between 3-4 of 10 "simultaneous" requests. I guess, it's triggering 403 forbbiden , cause the http session is detected as a new one, without any previous security applied, and without session attributes neither.

Temporal solution

In a request filter (before RESTful service), it checks authorization with isUserInRole(). If it fails, it makes a check in database.

If you have any other question, I'll be here.

@OndroMih
Copy link
Contributor

OndroMih commented Dec 6, 2024

Note that I was able to reproduce the issue as described in the description: #25141 (comment).

I was able to reproduce it when I logged in with the same user at the same time in multiple browser sessions. The I started seeing 403 responses from time to time. When I logged in only once, I didn't see any 403 response, or maybe I did, but much less frequently.

@Isaacrgz
Copy link

Isaacrgz commented Dec 8, 2024

Note that I was able to reproduce the issue as described in the description: #25141 (comment).

I was able to reproduce it when I logged in with the same user at the same time in multiple browser sessions. The I started seeing 403 responses from time to time. When I logged in only once, I didn't see any 403 response, or maybe I did, but much less frequently.

I made a project to reproduce 403 response easily.

@OndroMih
Copy link
Contributor

OndroMih commented Dec 9, 2024

Thanks. I deployed it to our cloud and reproduced it here: http://node11108-gf-25141.node.cloudlets.zone:8080/

@Isaacrgz
Copy link

Sets of 5 requests

Hi @OndroMih, testing the app in your cloud, I did 4 sets of 5 requests each.

  • The first ´403´ from the 3º set was triggered 2 seconds (approx) after the last request from the 2º set.
  • The first request from the 4º set was too close to the previous set (less than a second).

I hope this can help.

@OndroMih
Copy link
Contributor

Hi,

I reproduced this on all versions of GlassFish since 7.0.0, with a few patches on older versions because the app wouldn't deploy. So there's no obvious regression. We'll need to look more deeply what's happening, probably enable some detailed logging.

To reproduce with a script, I did the following:

Increase the maximum allowed connetions and threads:

set configs.config.server-config.thread-pools.thread-pool.http-thread-pool.max-thread-pool-size=10000
set configs.config.server-config.network-config.protocols.protocol.http-listener-1.http.max-connections=10000
set configs.config.server-config.network-config.transports.transport.tcp.max-connections-count=10000

And then the following using the curl tool (using the ZSH shell):

STATUS_FILE=/tmp/status
echo OK > $STATUS_FILE
curl -c /tmp/curl-cookies -X POST http://localhost:8080/JakartaSecurity-1.1/servlet
for ((i = 1; i < 10000; i++)); do
    (curl -f -b /tmp/curl-cookies -X GET http://localhost:8080/JakartaSecurity-1.1/resources/service/protected || echo FAIL > $STATUS_FILE) &
    pids[${i}]=$!
done

if /tmp/status contains OK when the script is finished, it means all the 10000 requests ended OK. If it contains FAIL, then one of them ended with 403.

@OndroMih
Copy link
Contributor

After debugging, I found out that the 403 response is given because in EEInstanceListener.handleAfterEvent, invocationManager.getCurrentInvocation() == null (

if (invocationManager.getCurrentInvocation() == null) {
), and then a call to realm.logout will set the security context for the current thread to null, in 403 response.

I don't know what causes the current invocation to become null, I need to investigate further.

OndroMih added a commit to OndroMih/glassfish that referenced this issue Dec 17, 2024
The securityContext is stateful, cannot be shared among requests. 
RealmAdapter stores the principal from request to it. We need to create 
a copy of the default context for each request instead of 
using the shared default context.
@OndroMih OndroMih linked a pull request Dec 17, 2024 that will close this issue
@OndroMih
Copy link
Contributor

I found the issue. All is caused by saving principal from request it to a security context which is shared by multiple threads (the default security context). Later the principal is removed for one thread and thus is removed also for other threads that may still need it. Those other threads result in 403 because the current principal doesn't contain the Group (role) principal. More details in #25290

OndroMih added a commit to OndroMih/glassfish that referenced this issue Dec 19, 2024
This way the behavior is not modified for the default security context, which will remain a singleton. 
The session principal is stored in a thread local so that each request has its own value even if they 
share the same default security context.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants