Skip to content

Commit 872994e

Browse files
authored
chore: multiple fixes to e2e report (#560)
* chore: remove unused deno.json field When running deno, it warns that this is unused when also specifying `imports`. * chore: fix type errors in Deno files We weren't including Deno files in `tsc` typechecking, but we also weren't using any `deno` commands that typecheck. This fixes obscured errors. I wanted to introduce typechecking in CI in this commit, but it turned out harder than I expected, so I've pulled that into a separate PR. The JSON hack isn't great but really it's the whole "import a json and write new data to it" pattern that should be simplified to "just read the new data in memory", since there's no actual reason to write it to disk if you follow the code paths. It was just done that way because it was the shortest path to add the dynamic fissue annotation from the previous implementation. * chore(e2e-report): fix handling of skipped suites Some parts of the code weren't correctly handling the case where an entire suite is skipped, with no specific skipped tests specified. 1. The `testCount.skipped` before this fix was strangely the number of specifically skipped tests plus the total number of suites which are either entirely skipped or contain skipped tests (i.e. it was pretty wrong). 2. The computed `skipped` count within each test suite was also wrong but in a different way. It was not counting at all skipped tests where the entire suite is skipped. I've also commented some existing correct logic that is very unintuitive. * chore(e2e): split open issues and skipped tests It was really confusing, both as a person viewing the page, and as a developer working with the code. This code and this component were muddling three different data types: failed tests, skipped tests, and skipped test suites. It was really error prone and hard to follow and modify. I've split this into separate components and sections on the page for failed tests ("open issues") and skipped tests, and I've passed skipped tests and skipped suites separately into the latter component. I've also refactored some code with side effects to be easier to work with. * refactor(e2e): remove stale todo * chore(e2e): update e2e test-results.json fixture This is the latest run on "latest" against main, from last night. * chore: remove unused deno scripts * chore(e2e): add test-results.json fixture to second path We expect this in two different locations depending on the code path... Having a sample file here allows for TypeScript to infer the type correctly. * chore(e2e): add back some skipped tests we removed The tests were named on canary, but we still need the old test names for `latest`. * chore(e2e): fix bad skipped test name, skip 2 more * chore(e2e): add missing tests to e2e-skip-retry.json * chore(e2e): add one more skipped test * chore(e2e): fix appending of skipped suites It was appending entirely skipped suites AND partially skipped suites. * chore(e2e): update test-results.json fixtures again * chore(e2e): add sample issues.json for local dev
1 parent aefcf63 commit 872994e

19 files changed

+13715
-5478
lines changed

.gitignore

+1-2
Original file line numberDiff line numberDiff line change
@@ -5,17 +5,16 @@ edge-runtime/vendor
55

66
# Local Netlify folder
77
.netlify
8+
89
/test-results/
910
/playwright-report/
1011
/blob-report/
1112
/playwright/.cache/
1213

1314
deno.lock
1415
.eslintcache
15-
/report/index.html
1616
.DS_Store
1717
tests/**/package-lock.json
1818
tests/**/pnpm-lock.yaml
1919
tests/**/yarn.lock
2020
tests/**/out/
21-
report/test-results.json

deno.json

+4-3
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,12 @@
11
{
22
"lint": {
33
"files": {
4-
"include": ["edge-runtime/middleware.ts"]
4+
"include": [
5+
"edge-runtime/middleware.ts"
6+
]
57
}
68
},
79
"imports": {
810
"@netlify/edge-functions": "https://edge.netlify.com/v1/index.ts"
9-
},
10-
"importMap": "./edge-runtime/vendor/import_map.json"
11+
}
1112
}

e2e-report/app/globals.scss

+23-17
Original file line numberDiff line numberDiff line change
@@ -68,7 +68,6 @@ span {
6868

6969
p {
7070
font-size: $base-font-size;
71-
font-weight: bold;
7271
}
7372

7473
img {
@@ -92,10 +91,6 @@ th {
9291
font-weight: 400;
9392
}
9493

95-
tr {
96-
font-weight: bold;
97-
}
98-
9994
a:visited,
10095
a:link {
10196
color: white;
@@ -241,14 +236,14 @@ button.nav {
241236
overflow-x: hidden;
242237
}
243238

244-
.skipped.card:hover {
239+
.skipped.card:hover,
240+
.open-issue.card:hover {
245241
background-color: initial;
246242
color: initial;
247243
}
248244

249245
.testCases {
250246
display: none;
251-
// background: $netlify-blue;
252247
padding: 0.5% 2%;
253248
box-shadow: inset -1px 4px 10px #00000047;
254249
border-radius: 11px;
@@ -313,15 +308,17 @@ button.nav {
313308
}
314309
}
315310

316-
.card.skipped {
311+
.card.skipped,
312+
.card.open-issues {
317313
display: none;
318314
width: clamp(350px, 55%, 1000px);
319315
cursor: default;
320316
height: 70vh;
321317
overflow: scroll;
322318
}
323319

324-
.card.skipped.open {
320+
.card.skipped.open,
321+
.card.open-issues.open {
325322
display: block;
326323
}
327324
}
@@ -365,7 +362,7 @@ button.nav {
365362
}
366363
}
367364

368-
.card:hover:not(.skipped) {
365+
.card.test-group:hover {
369366
background-color: $netlify-blue;
370367
background-image: $card-bg-img;
371368
background-blend-mode: screen;
@@ -400,7 +397,8 @@ span[data-status='skipped'] {
400397
width: 35vw;
401398
}
402399

403-
.skipped {
400+
.skipped,
401+
.open-issues {
404402
td,
405403
th {
406404
padding: 0.5% 2%;
@@ -412,19 +410,26 @@ span[data-status='skipped'] {
412410
font-size: clamp(1rem, 0.8764rem + 0.4121vw, 1.16rem);
413411
}
414412
}
413+
}
415414

415+
.open-issues {
416416
a {
417417
color: black;
418418
font-weight: bold;
419-
display: flex;
420-
flex-flow: column;
419+
display: inline;
421420
}
422-
:hover a {
423-
color: white;
421+
a:hover {
422+
color: $netlify-blue;
423+
}
424+
.github-link-icon {
425+
width: 0.75em;
426+
height: auto;
427+
margin: 0 0.5em;
424428
}
425429
}
426430

427-
.skipped .card {
431+
.skipped .card,
432+
.open-issues .card {
428433
display: flex;
429434
justify-content: space-between;
430435
padding: 4%;
@@ -433,7 +438,8 @@ span[data-status='skipped'] {
433438
}
434439
}
435440

436-
.skipped .card:hover {
441+
.skipped .card:hover,
442+
.open-issues .card:hover {
437443
p {
438444
color: #2bdcd2;
439445
}

e2e-report/app/page.js

+10-15
Original file line numberDiff line numberDiff line change
@@ -1,23 +1,17 @@
1-
import { SkippedTests } from '../components/filter-data.js'
1+
import { OpenIssues, SkippedTests } from '../components/filter-data.js'
22
import GroupedTests from '../components/grouped-tests.js'
33
import Hero from '../components/hero.js'
44
import testData from '../data/test-results.json'
55

66
export default function Home() {
77
const { results, passed, failed, total, passRate, skipped, testDate, nextVersion } = testData
8-
const skippedTests = []
9-
results.forEach((suite) => {
10-
if (suite.skipped === true) {
11-
skippedTests.push(suite)
12-
}
13-
14-
const { testCases } = suite
15-
testCases?.forEach((testCase) => {
16-
if (testCase.status === 'failed') {
17-
skippedTests.push(testCase)
18-
}
19-
})
20-
})
8+
const skippedSuites = results.filter(({ skipped }) => skipped === true)
9+
const skippedTestCases = results.flatMap(
10+
({ testCases }) => testCases?.filter(({ status }) => status === 'skipped') ?? [],
11+
)
12+
const failedTestCases = results.flatMap(
13+
({ testCases }) => testCases?.filter(({ status }) => status === 'failed') ?? [],
14+
)
2115

2216
return (
2317
<>
@@ -34,7 +28,8 @@ export default function Home() {
3428
</div>
3529
<div className="grid">
3630
<GroupedTests testData={results} />
37-
<SkippedTests testSuites={skippedTests} />
31+
<OpenIssues testCases={failedTestCases} />
32+
<SkippedTests testSuites={skippedSuites} testCases={skippedTestCases} />
3833
</div>
3934
</>
4035
)

e2e-report/components/filter-data.js

+74-15
Original file line numberDiff line numberDiff line change
@@ -4,6 +4,7 @@ import { useState } from 'react'
44

55
import Down from '../public/down.svg'
66
import Up from '../public/up.svg'
7+
import ExternalLinkIcon from '../public/arrow-up-right-from-square-solid.svg'
78

89
export const groupDefinitions = [
910
{
@@ -58,7 +59,60 @@ export const groupTests = (testSuites) => {
5859
)
5960
}
6061

61-
export const SkippedTests = ({ testSuites }) => {
62+
export const OpenIssues = ({ testCases }) => {
63+
const [slider, setSlider] = useState({})
64+
65+
function handleSelect(el) {
66+
setSlider({
67+
...slider,
68+
[el]: !slider[el],
69+
})
70+
}
71+
72+
return (
73+
<div className="wrapper">
74+
<div className="card" onClick={() => handleSelect('openIssues')}>
75+
<h4>Open Issues</h4>
76+
<p>Total: {testCases.length}</p>
77+
{slider.openIssues ? (
78+
<Up className="arrow" width={200} height={150} />
79+
) : (
80+
<Down className="arrow" width={200} height={150} />
81+
)}
82+
</div>
83+
<table className={`testSuite card open-issues ${slider.openIssues ? 'open' : 'close'}`}>
84+
<tbody>
85+
<tr>
86+
<th>Test</th>
87+
<th>Reason</th>
88+
</tr>
89+
{testCases.map((testCase, index) => {
90+
const { name, link, reason = 'Reason not yet assigned' } = testCase
91+
return (
92+
<tr key={index}>
93+
<td>{name}</td>
94+
<td>
95+
<p>
96+
{link ? (
97+
<a href={link} target="_blank">
98+
<ExternalLinkIcon className="github-link-icon" />
99+
{reason}
100+
</a>
101+
) : (
102+
reason
103+
)}
104+
</p>
105+
</td>
106+
</tr>
107+
)
108+
})}
109+
</tbody>
110+
</table>
111+
</div>
112+
)
113+
}
114+
115+
export const SkippedTests = ({ testCases, testSuites }) => {
62116
const [slider, setSlider] = useState({})
63117

64118
function handleSelect(el) {
@@ -71,8 +125,10 @@ export const SkippedTests = ({ testSuites }) => {
71125
return (
72126
<div className="wrapper">
73127
<div className="card" onClick={() => handleSelect('skipped')}>
74-
<h4>Open Issues + Skipped Tests</h4>
75-
<p>Total: {testSuites.length}</p>
128+
<h4>Skipped Tests</h4>
129+
<p>
130+
Total: {testSuites.length} suites + {testCases.length} tests
131+
</p>
76132
{slider.skipped ? (
77133
<Up className="arrow" width={200} height={150} />
78134
) : (
@@ -85,21 +141,24 @@ export const SkippedTests = ({ testSuites }) => {
85141
<th>Test</th>
86142
<th>Reason</th>
87143
</tr>
88-
{testSuites?.map((testCase, index) => {
89-
const { name, link, reason, file } = testCase
144+
{testSuites.map((testCase, index) => {
145+
const { file, reason } = testCase
90146
return (
91-
<tr key={`${index}`}>
92-
<td>{file || name}</td>
147+
<tr key={index}>
148+
<td>{file}</td>
149+
<td>
150+
<p>{reason}</p>
151+
</td>
152+
</tr>
153+
)
154+
})}
155+
{testCases.map((testCase, index) => {
156+
const { name, reason } = testCase
157+
return (
158+
<tr key={index}>
159+
<td>{name}</td>
93160
<td>
94161
<p>{reason}</p>
95-
{link && (
96-
<button className="github">
97-
<a href={link} target="_blank">
98-
{' '}
99-
Github Issue
100-
</a>
101-
</button>
102-
)}
103162
</td>
104163
</tr>
105164
)

e2e-report/components/grouped-tests.js

+1-1
Original file line numberDiff line numberDiff line change
@@ -35,7 +35,7 @@ export default function GroupedTests({ testData }) {
3535
const groupTotal = (testGroup.passed ?? 0) + (testGroup.failed ?? 0)
3636
return (
3737
<div key={testGroup.id} className="wrapper">
38-
<div className="card" onClick={() => handleSelect(testGroup.id)}>
38+
<div className="card test-group" onClick={() => handleSelect(testGroup.id)}>
3939
<Table
4040
th={['Test suite type:', '# of suites:', '# of tests:', 'Passing:']}
4141
name={testGroup.title}

e2e-report/components/test-suites.js

+1-5
Original file line numberDiff line numberDiff line change
@@ -1,11 +1,7 @@
11
import Table from './table.js'
22

33
export default function TestSuites({ suite, slider, handleSelect, arrows }) {
4-
const { name, file, passed, failed, skipped, testCases } = suite
5-
// TODO: There are some suites coming through that have no status and some repeats
6-
if (skipped === true || (passed === 0 && failed === 0 && skipped === 0)) {
7-
return
8-
}
4+
const { name, file, passed, failed, testCases } = suite
95

106
return (
117
<>

e2e-report/data/issues.json

+46
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,46 @@
1+
[
2+
{
3+
"body": "I'm not quite sure exactly what the scope of this is, but the failing tests here have a fixture site that enables `skipMiddlewareUrlNormalize` and a user middleware that redirects any other casing of `/en/*` to `/en/*` (and so on), but when `/EN` is fetched it redirects to `/en/en` instead of `/en`:\r\n\r\n```\r\n ● skip-trailing-slash-redirect › should be able to redirect locale casing $1\r\n\r\n expect(received).toBe(expected) // Object.is equality\r\n\r\n Expected: \"/en\"\r\n Received: \"/en/en\"\r\n\r\n 155 | const res = await next.fetch(`/${locale}`, { redirect: 'manual' })\r\n 156 | expect(res.status).toBe(307)\r\n > 157 | expect(new URL(res.headers.get('location'), 'http://n').pathname).toBe(\r\n | ^\r\n 158 | `/${locale.toLowerCase()}`\r\n 159 | )\r\n 160 | }\r\n\r\n at toBe (e2e/skip-trailing-slash-redirect/index.test.ts:157:73)\r\n\r\n ● skip-trailing-slash-redirect › should be able to redirect locale casing $1\r\n\r\n expect(received).toBe(expected) // Object.is equality\r\n\r\n Expected: \"/ja-jp\"\r\n Received: \"/ja-jp/ja-jp\"\r\n\r\n 155 | const res = await next.fetch(`/${locale}`, { redirect: 'manual' })\r\n 156 | expect(res.status).toBe(307)\r\n > 157 | expect(new URL(res.headers.get('location'), 'http://n').pathname).toBe(\r\n | ^\r\n 158 | `/${locale.toLowerCase()}`\r\n 159 | )\r\n 160 | }\r\n\r\n at toBe (e2e/skip-trailing-slash-redirect/index.test.ts:157:73)\r\n```\r\n\r\nIt looks like we [previously claimed to have fixed this in FRA-332](https://github.com/netlify/next-runtime-minimal/pull/287), but it must have regressed at some point.\r\n\r\ntest: test/e2e/skip-trailing-slash-redirect/index.test.ts\r\nreason: does not correctly handle user middleware that redirects to path with canonical locale casing when app enables `skipMiddlewareUrlNormalize` and path contains locale slug with non-canonical casing",
4+
"number": 564
5+
},
6+
{
7+
"body": "A pages router site with both basepath and i18n enabled does not match middleware targeted at the root unless a locale is in the URL. It should match when using the default locale\r\n\r\nDemo: \r\n- https://6634b69452829cd391cab206--next-e2e-tests.netlify.app/root/ does not have `x-from-middleware` header\r\n- https://6634b69452829cd391cab206--next-e2e-tests.netlify.app/root/en/ does have `x-from-middleware` header\r\n\r\ntest: test/e2e/middleware-matcher/index.test.ts\r\nreason: Middleware does not match when using basePath and default locale",
8+
"number": 454
9+
},
10+
{
11+
"body": "There are a few scenarios where our the custom matchers from middleware are not correctly being transformed when using i18n and default locales, as they're either matching too broadly or too narrowly. See the two linked GH issues for details:\r\n\r\n[https://github.com/netlify/next-runtime-minimal/issues/453](https://github.com/netlify/next-runtime-minimal/issues/453)<br>[https://github.com/netlify/next-runtime-minimal/issues/454](https://github.com/netlify/next-runtime-minimal/issues/454)\r\n\r\ntest: Middleware custom matchers i18n should not match\r\nreason: Middleware matching is too broad when using i18n",
12+
"number": 453
13+
},
14+
{
15+
"body": "On sites that use pages router and have middleware, loading a page using next/link will attempt to load a JSON file, which will return a 404. If there is no middleware then it works fine. This applies even if the middleware does nothing.\r\n\r\ntest: test/e2e/middleware-base-path/test/index.test.ts\r\nreason: Pages router data requests returning 404 when middleware is used\r\ntest case: \"Middleware base tests router.query must exist when Link clicked page routing\"",
16+
"number": 450
17+
},
18+
{
19+
"body": "When appending a `set-cookie` header, the server returns two copies of the header. e.g. \r\n\r\n```ts\r\nexport async function middleware(request, ev) {\r\n const next = NextResponse.next()\r\n next.headers.append('set-cookie', 'bar=chocochip')\r\n return next\r\n}\r\n```\r\n\r\nLeads to: \r\n\r\n![image](https://github.com/netlify/next-runtime-minimal/assets/213306/22c51860-fbde-4b70-b9aa-18172ee0d6e2)\r\n\r\ntest: test/e2e/middleware-responses/test/index.test.ts\r\nreason: Appending set-cookie header in middleware leads to duplicate header\r\n",
20+
"number": 447
21+
},
22+
{
23+
"body": "The test [\"app-dir action handling fetch actions should store revalidation data in the prefetch cache\"](https://github.com/vercel/next.js/blob/6475431a4cbbf2b71c38158e0e722183779faf4f/test/e2e/app-dir/actions/app-action.test.ts#L980) fails when running tests, but seems to work fine when testing manually. It may be a first-run issue, which needs investigation.\r\n\r\ntest: test/e2e/app-dir/actions/app-action.test.ts\r\nreason: Fetch action prefetch cache test is flakey",
24+
"number": 444
25+
},
26+
{
27+
"body": "If a Next.js page returns a 500 error and the browser sent accept-encoding gzip, it seems the returned data _is_ encoded, but no encoding header is set, so the browser cannot decode it.\r\n\r\n`curl --request GET --url https://66056f2be8186e00aaea53d3--next-e2e-tests.netlify.app/enoent --header 'Accept-Encoding: gzip'`\r\n\r\ntest case: https://github.com/vercel/next.js/blob/canary/test/e2e/getserversideprops/test/index.test.ts#L367\r\ntest: test/e2e/getserversideprops/test/index.test.ts\r\nreason: Server error pages return encoded data without content-encoding header if accept-encoding is gzip",
28+
"number": 387
29+
},
30+
{
31+
"body": "When redirecting a data request, middleware returns a response with `x-nextjs-redirect`, rather than a `location` header. We handle this correctly. However Next.js expects us to directly return and empty response with a 302 response code (without the location header), whereas we're currently passing the request on to the origin and returning the body with 404 code. I'm unsure if it's legal to return a 302 with no location, but it's what next start does, and the router expects.\r\n\r\ntest case: https://github.com/vercel/next.js/blob/canary/test/e2e/middleware-redirects/test/index.test.ts#L100\r\ntest: test/e2e/middleware-redirects/test/index.test.ts\r\nreason: Pages router middleware should return 302 status for redirected data requests",
32+
"number": 386
33+
},
34+
{
35+
"body": "In sites with trailing slashes enabled, requests should not add a trailing slash when they're for non-data static files.\r\n\r\ntest case: https://github.com/vercel/next.js/blob/canary/test/e2e/middleware-trailing-slash/test/index.test.ts#L436-L437\r\ntest: test/e2e/middleware-trailing-slash/test/index.test.ts\r\nreason: Middleware should not add trailing slashes to non-data requests in static dir",
36+
"number": 385
37+
},
38+
{
39+
"body": "Rewrites in middleware on i18n sites add the locale to the target automatically. However it is supposed to skip this if the target is a static file. This does not currently work, as the middleware doesn't know if a static file exists. This is documented as a known issue.\r\n\r\ntest: test/e2e/i18n-ignore-rewrite-source-locale/rewrites-with-basepath.test.ts, test/e2e/i18n-ignore-rewrite-source-locale/rewrites.test.ts\r\nreason: Middleware on sites with i18n cannot rewrite to static files",
40+
"number": 383
41+
},
42+
{
43+
"body": "It doesn't seem to be documented, but the e2e test and fixture here expects a CSP to be automatically applied to scripts in the head. The docs show manually setting it, but [the fixture](https://github.com/vercel/next.js/blob/canary/test/e2e/app-dir/app/middleware.js#L30) seems to only set it in the response. Nevertheless, running `next start` does seem to set it automatically, but when deployed it doesn't. I think this is low priority because it seems to be undocumented behaviour.\r\n\r\ntest case: https://github.com/vercel/next.js/blob/canary/test/e2e/app-dir/app/index.test.ts#L1711\r\ntest: test/e2e/app-dir/app/index.test.ts\r\nreason: Nonce not automatically set in script tags when using CSP",
44+
"number": 381
45+
}
46+
]

0 commit comments

Comments
 (0)